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REPORT TO THE CONGRESS 



BY THE COMPTROLLER GENERAL 
OF THE UNITED STATES 



The National Assessment 
Of Educational Progress: 
Its Results Need To 
Be Made More Useful 

National Center for Education Statistics 
Department of Health, Education, and Welfare 



The National Assessment of Educational Prog- 
ress is a project which annually surveys the 
knowledge, skills, and attitudes of young 
Americans. I ts basic mission is to provide 
information useful to educational decision- 
makers and practitioners. The project has con- 
tributed to American education, but its assess- 
ment results have been of limited usefulness. 

To make the project's results more useful, the 
Secretary of Health, Education, and Welfare 
should redirect the project by (1) identifying 
informational and other needs of decision- 
makers, (2) determining the feasibility and 
cost effectiveness of alternative approaches to 
satif.fy those ncfids, and (3) deciding on tho 
assessment approach to be used. 
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To the President of the Senate and the 
Speaker of the House of Representatives 

The National Assessment of Educational Progress is a 
project supported through contracts with the National Cen- 
ter for Education Statistics, Department of Health, Educa- 
tion, and Welfare. The project assesses the knowledge, 
skills, and attitudes of selected age groups of Americans 
to provide information useful to educational decisionmakers 
and practitioners . 

This report points out that, although the National As- 
sessment has contributed to American education, its assess- 
ment results have been of limited usefulness. The report 
includes recommendations to make the assessment results 
more useful. 

We made our review pursuant to the Budget and Account- 
ing Act, 1921 (31 U.S.C. 53), and the Accounting and Audit- 
ing Act of 1950 (31 U.S.C. 67). 

We are sending copies of this report to the Director^ 
Office of Management and Budget, and the Secretary of Health, 
Education, and Welfare. A 




Comptroller General 
of the United States 
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DIGEST 



How much good is the yearly expenditure of 
billions of dollars on education doing in 
terms of what Americans know and can do? 

The National Assessment of Educational Pro- 
gress is a project which, annually since 
1969, has tried to provide answers to this 
question through surveys of the knowledge, 
skills, and attitudes of selected age groups 
of young Americans. Its basic goal is to 
provide information useful to educational 
decisionmakers and practitioners. (See 
p. 1.) 

The project grew from a realization in the 
early 1960s, when the Federal Government 
began investing heavily in formal education, 
that no comprehensive, dependable informa- 
tion existed on the educational attainments 
of Americans. 



National Assessment officials believe that 
the project's purpose is related to the move- 
ment toward accountability in education. 
This movement emphasizes that the purpose of 
schools is to provide education and seeks to 
hold education officials accountable for this. 
To evaluate education, one needs information 
about the knowledge and skills of the student 
population. (See p. 3.) 

The National Assessment, located in Denver, 
has been a project of the Education Commis- 
sion of the States since 1969. The Commis- 
sion is a nonprofit organization formed by 
interstate compact in 1966. (See p. 2.) 
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The project has received about $35.5 million 
in Federal support through f-iscal year 1976, 
It has contributed to American education 
by advancing educational assessment technology 
and helping State education agencies to apply 
assessment technology. However > National As- 
sessment results have been of limited useful- 
ness to education decisionmakers, researchers, 
and practitioners. This has prevented the 
project from achieving its basic goal. (See 
pp. 3, 10/ 23, and 25.) 

To make the project's results more useful, 
GAO is recommending that the Secretary of 
Health, Education, and Welfare redirect the 
project by 

— identifying the informational and other 
needs of decisionmakers, 

— determining the feasibility and cost effec- 
tiveness *of alternative approaches for col- 
lecting and reporting educational assess- 
ment data and providing other services to 
satisfy those needs, and 

— deciding on the assessmen approach to be 
used . ( See p. 34 . ) 

Although consultation with project officials 
seems necessary during this process, it is 
also important to seek the views of users 
or potential users of project results at 
national. State, and local levels and of 
recognized experts in educational manage- 
ment, assessment, and research. (See pp. 34 
and 35.) 

Examples of possible alternative strategies 
for the National Assessment include: 

— Restricting the project's data collection 
efforts and primarily providing assessment 
models, test objectives and questions, and 
technical assistance concerning assessment 
to States, local school districts, etc. 

— Expanding the project's data collection 
efforts to provide more information on geo- 
graphic, demographic, or background vari- 
ables . 
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—Focusing data collection efforts on the 
information needs of one level of govern- 
ment, such as the Federal level, and re- 
structuring subject areas, sample size, 
and demographic and background variables 
to serve those needs within a shorter time 
frame . 

—Making more modest changes in the present 
mixture or manner of providing technical 
assistance, planned assessment data, and 
special studies. (see p. 35.) 

The Secretary should require project man- 
agement to increase the use and improve 
the usefulness of project data by 

—Establishing continuous dialogues with the 
Congress, executive agencies, and State 
and local policymakers to determine their 
data needs and how the National Assessment 
can best meet those needs. 

—Continuing and increasing recent efforts 
to interpret project data so as to rely 
less on others for data interpretation. 

—Providing for comparison of its test re- 
sults with performance standards by givina 
greater emphasis to developing the pro- 
cedures needed to compare those results to 
generally acceptable non-Federal standards. 

—Improving its communication and cooperation 
oi^L I National Institute of Education and 
other educational researchers to facilitate 
possible research, interpretation, and ap- 
plications of project results. 

""fsee°p^"f5'^)^^^'"^"^^^°" °^ project results. 

GAO is also recommending that the Secretary 
urge project management to better use the 
project's capabilities by 

—Continuing to develop and refine objective- 
referenced tests— which give results for a 
obje?tives!""^ °^ predetermined educational 
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— Increasing its technical assistance to 
local education agencies regarding its 
concepts, materials, and methods when 
possible within the available funding. 

— Improving its services to Federal agen- 
cies and others by providing special 
studies for those who need and are will- 
ing to finance them. Accordingly, the 
project should better inform potential 
users of its capabilities. (See p. 21.) 

The Department of Health, Education, and 
Welfare stated that it is reexamining the 
project to decide on the necessity, nature, 
or extent of any redirection. It does not 
believe redirection is warranted until the 
reexamination has been completed. 

Regarding GAO's recommendation that the 
Department require the project to give 
greater emphasis to developing the pro- 
cedures needed to compare its test re- 
sults to generally acceptable standards, 
the Department stated that it will (1) 
explore the standards problem with the 
project's staff and (2) encourage efforts 
to provide the technical and methodologi- 
cal guidance needed for policymakers at 
various levels to make their own com- 
parisons or judgments concerning appro- 
priate standards. The project went 
beyond this and agreed to provide for 
comparison of its test results with per- 
formance standards. (See p. 36.) 

Lack of performance standards is an im- 
porant factor limiting the usefulness of 
the project's data. However, the project's 
commitment to provide for comparison with 
standards is a good first step toward solv- 
ing the problem. (See p. 36.) 

The Department agreed with GAO's other rec- 
ommendations, and the National Assessment 
agreed with all of GAO's recommendations. 
Both the Department and the National Assess 
ment described actions taken or planned to 
implement the recommendations. (See pp. 21 
22, 36, and 37.) 
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CHAPTER 1 



INTRODUCTION 



^rhe National Assessment of Educational Progress is a 
project which annually surveys the knowledge, skills, and 
attitudes of selected age groups of young Americans. The 
project, which began as an exploratory committee in 1964, 
made its first assessments in 1969. 

The National Assessment seeks to help answer this ques- 
tion: How much good is the yearly expenditure of billions 
of dollars on education doing in terms of what Americans 
know and can do? Its basic goal, as stated in its 5-year 
plan, is "to provide information useful to educational deci- 
sionmakers and practitioners in identifying problems, setting 
priorities, and determining progress." This is to be accom- 
plished by achieving the following eight goals: 

— Measuring changes in knowledge and skills of young 
Americans over time. 

— Acquiring data on educational attainments. 

— Improving the technology needed for gathering and 
analyzing National Assessment achievement data. 

— Facilitating the use of National Assessment techno- 
logy at the State and local levels. 

— Making special studies of selected areas of educa- 
tional attainment. 

— Promoting interpretations of National Assessment 
data. 

— Disseminating findings. 

— Conducting appropriate research. 

The National Assessment's report on its goals and accom- 
plishments from 1969 through 1975 indicates that the collec- 
tion of concrete information — such as comprehensive data on 
educational attainments and the measurement of changes in 
these attainments—should, over time, assist national educa- 
tional policymakers in decisionmaking. According to its 5- 
year plan, the project also seeks to provide information to 
State and local educational decisionmakers, curriculum devel- 
opers, and researchers. 
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A National Assessment information booklet states that 
the project's reports should provide many ideas for local and 
Sta^e assessment projects, curriculum design, instructional 
materials development, and teacher education. 

The National Assessment, located in Denver, has been a 
project of the Education Commission of the States since 1969. 
The Commission is a nonprofit organization formed by inter- 
state compact in 1966. Forty-five States, Puerto Rico, and 
the Virgin Islands are members. Their representatives in- 
clude Governors, chief State school officers, and legisla- 
tors. The Commission's purpose is to discuss mutual educa- 
tional problems and act together to achieve common goals. 

The National Assessment operates under annual contracts 
with the National Center for Education Statistics in the Of- 
fice of the Assistant Secretary for Education, Department of 
Health, Education, and Welfare (HEW). Before fiscal year 
1975, HEW's Office of Education awarded annual contracts to 
the National Assessment. The current contract supports the 
National Assessment goals and specifies the tasks to be per- 
formed for achieving those goals. The National Center for 
Education Statistics 

— reviews the contract proposal and subcontracts; 

— coordinates the award of the contract with HEW con- 
tracting specialists; and 

— broadly monitors the contract through such means as 
providing policy guidance to the project, participa- 
ting in the deliberations of various project policy 
and advisory committees, and reviewing materials being 
developed for publication. 

NEED FOR THE NATIONAL ASSESSMENT 

During the past decade, the Nationa] Assessment devel- 
oped from an idea shared by a small group of distinguished 
scholars and outstanding educational practitioners to an on- 
going operational project. It has about 80 staff members, 
utilizes major subcontractors, and tests thousands of individ- 
uals each year. 

The project grew from a realization in the early 1960s, 
when the Federal Government began investing billions of dol- 
lars annually in formal education, that no comprehensive and 
dependable information existed about the educational attain- 
ments of Americans. The only available measures of educa- 
tional quality were based on such information as the number 
of classrooms, teacher-student ratios, and dollars spent per 
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student. No significant direct assessment had been made of 
educational outcomes — what students do or do not know and can 
or cannot do. Although State- or school-administered stand- 
ardized tests, which provided scores to compare students^ 
were useful for categorizing students^ they were not designed 
to provide information about what students were actually 
learning . 

National Assessment officials believe that the project's, 
purpose is related to the current movement toward accountabi- 
lity in education. Educational accountability emphasizes 
that the purpose of schools is to provide education, and it 
seeks to hold education officials accountable for educating 
children. To evaluate children's education, one needs infor- 
mation aDout the knowledge and skills of the student popula- 
tion. 

The idea of a National Assessment encountered consider- 
able early resistance, principally among the States, because 
the project was perceived as a nationwide testing program 
providing comparative data on a State-by-State basis. These 
concorns are reflected in the fact that the National Assess- 
ment reports data only on national and regional bases to pro- 
vide more neutral and general data. 

FUNDING 

The project, including preliminary committees estab- 
lished to explore its feasibility and plan its approach, has 
been supported by both private and public funds since 1964. 
Through fiscal year 1970, private foundations provided nearly 
$4.5 million to support its development. Federal support 
through the Office of Education and the National Center for 
Education Statistics has amounted to about $35.5 million 
through fiscal year 1976. Funding by year and source is 
shown on the following page. 
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Source 

Carnegie Corporation 
Ford Foundation 
Office of Education 
National Center for 
Education Statistics 



Total 



Fiscal 
years (note a ) 

1964-1970 
b/1966-1969 
1968-1976 

1975-1976 



Funding 

$ 2,782,900 
1,696,000 
c/26,124,394 

d/ 9,400,000 

$40,003,294 
e/ 100,000 

$40,103,294 



a/The National Assassment ' s fiscal year is October 1 
through September 30. 

b/No Ford Foundation funding was provided for fiscal year 
1968. 

c/Includes $352,036 from the Right to Read program for as- 
sessing functional literacy in fiscal years 1974-76. No 
other Office of Education funds were provided for fiscal 
years 1975 and 1976. 

d/Includes fiscal year 1976 funding of $4.9 million. 

e/National Assessment funds derived from Office of Educa- 
tion grants to the University of Minnesota in 1966 and 
1967. 

SCOPE OF REVIEW 

Because of the Federal investment in the National As- 
sessment, we were concerned with the project's usefulness 
and how it might be improved. 

To obtain State and local education agency views on the 
usefulness of the National Assessment and on related matters, 
we sent questionnaires to education agencies in all States 
and the District of Columbia and to a stratified statistical 
sample of local school districts throughout the Nation. The 
questionnaires were sent in April 1975 and were returned by 
June 1975. 

The District of Columbia and all but one State responded 
to our State-level questionnaire. To simplify reporting of 
questionnaire results in this report, we consider the Dis- 
trict of Columbia to be a State. State-level respondents 
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were nearly always officials responsible for statewide as- 
sessment, accountability, and/or testing activities* (See 
app. I for the compilation of responses on the State educa- 
tion agency questionnaire.) 

Our questionnaire sample for local school districts was 
largely the same as a national sample used by the Office of 
Education in 1973. Neither sample included any school dis- 
tricts having fewer than 300 pupils. Both were stratified 
according to enrollment as follows: 

125,000 pupils or more 

35,000 to 124,999 pupils 

9,000 to 34,999 pupils 

3,000 to 8,999 pupils 

300 to 2,999 pupils 

Nineteen school districts compose the first group — the 
largest school districts — and all were included in the sam- 
ple. An independent random sample of 813 school districts 
was drawn from the remaining groups. We received responses 
from 710 (85 percent) of the 832 school districts included in 
the sample. 

As a result of the high response rate, the attitudes and 
opinions expressed in response to our local questionnaire are 
representative of the universe of 11,666 local school dis- 
tricts — all those in the Nation having 300 or more pupils. 
However, we projected a total of 8,941 local education agen- 
cies responding because, based on the weighting and the re- 
sponse rates across the various strata in our sample, this 
method allows us to obtain the most accurate percentage 
breakdowns on the answers given. 

Most local respondents were directors of testing, but 
some superintendents responded. (For local education agency 
questionnaire results, see app. II. The numbers shown there 
represent the number of local school districts in the Nation 
to which our local questionnaire sample responses have been 
projected. ) 

To supplement information obtained from the question- 
naire, we interviewed officials of 5 State departments of 
education, the District of Columbia, and 10 local school dis- 
tricts. 

We reviewed policies and procedures and various reports 
prepared by and/or relating to the National Assessment. We 
interviewed officials of the National Assessment, the Educa- 
tion Commission of the States, the Office of the Secretary of 
HEW, the Office of the Assistant Secretary for Education, the 
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National Center for Education Statistics, the Office ^^^^q 
cation, the National Institute of Education, and the °^^pe^ 
of Managem'^nt and Budget* In addition, we interviewed 
staff of various congressional committees and officials 
education research organizations , including 4 commercial- ^^^t 
publishers, and 6 research/evaluation organizations, ^^^^^o 
interviewed officials of two national interest groups 
cerned with education. 



CHAPTER 2 

PROJECT INITIATION AND ADMINISTRATI ON 

During the early 1960s, the Commissioner of Education 
Became concerned about the lack of comprehensive and compar- 
able data on educational attainments. in 1963 he initiated 
a series of conferences to explore ways to provide this in- 
formation. The original 1867 Office of Education charter 
(14 Stat. 434) provided the authority for the Commissioner 
to determine the Nation's progress in education. 

PLANNING THE NATIONAL ASSESSMENT 

Project planning began in 1964 with establishment of an 
exploratory committee to .examine the possibilities of a na- 
tional assessment. Four years of work financed by the Car- 
negie Corporation and the Ford Foundation went into determin- 
ing how a national assessment could be designed, defining 
goals, developing measuring instruments, and developing a 
plan for conducting the assessment. This work was done in 
consultation with leading educators, subject matter special- 
ists, and interested laymen. 

The committee dc ;.\;d it was feasible to initiate a 
project to periodically assess the knowledge, understanding, 
skills, and attitudes in 10 learning areas (art, career and 
occupational development, citizenship, literature, mathema- 
tics, music, reading, science, social studies, and writing) 
at four age levels (9, 13, 17, and young adults aged 26-35). 
By 1969 a complete methodology and approach to collecting 
data had been designed, and the first assessment had begun in 
the areas of science, citizenship, and writing. 

These learning areas were selected because the National 
Assessment, together with its advisors and consultants, be- 
lieved in focusing on traditional subject matter areas rather 
than measuring skills and attitudes which go beyond these 
areas. Project officials perceived that national data on at- 
tainments m subject areas at different age levels and the 
changes in attainments over time could be of great interest 
to policymakers at national. State, and local levels. 

The age levels were selected to represent educational 
milestones that most students attain: age 9, when most stu- 
dents have been exposed to the basic program of primary edu- 
cation; age 13, when most students have completed their ele- 
mentary level education; and age 17, when most students are 
still m school and completing their secondary education. To 
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provide infoimation on all 17-year-olds, the project also 
sampled those out of school. In addition, young adults were 
sampled to assess the skills, knowledge, and attitudes of 
those who had completed their formal education. The samples 
are designed to enable the project to make inferences about 
the populations from which the samples are selected. 

As a result of recent budgetary restrictions, the pro- 
ject has at least temporarily discontinued the assessment of 
young adults and has reduced the 10 learning areas to 5: 
science, mathematics, reading, social studies and citizen- 
ship, and humanities (which includes literature, art, and 
music) . 

Data in each learning area is collected and reported on 
an overall national basis and by 

— four geographical regions (Northeast, Southeast, Cen- 
tral, and West); 

— seven sizes and types of community (extreme rural 
areas, extreme inner city, inner city fringe, urban 
fringe, affluent suburb, medium-sized city, and small 
city ) ; 

— four educational levels of parents (no more than 
eighth grade, more than eighth grade but less than 
high school graduation, high school graduation, and 
some formal education beyond high school); 

— race (black, white, and other); and 

— sex. 

To provide information on what the populations sampled 
are learning, the project designed and developed new test in- 
struments, referred to as objective-referenced tests. Stand- 
ardized norm-referenced achievement tests compare each stu- 
dent's performance with the average performance of other stu- 
dents. In contrast, the project's results are reported in 
terms of a percentage of the group tested that was able to 
perform an exercise or question. The sampling techniques do 
not require each respondent to answer all questions used in a 
learning area, and no respondent receives a score. Each ques- 
tion reflects a previously defined educational objective, 
which has been approved by a review group made up of educators 
and other citizens. 
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ADMINISTERING THE NATIONAL ASSESSMENT 



National Assessment staff are responsible for 

— developing individual test questions, 

— monitoring the administration and collection of field 
data as well as the sampling and scoring, 

— analyzing the data, 

— disseminating and reporting assessment results, and 

— providing technical assistance to States anca locali- 
ties interested in designing their own assessments. 

Subcontrators are responsible for drawing the sample, admin- 
istering the tests, collecting data, and scoring the results. 

Much of the project's work is concerned with Vear-by- 
year implementation of the overall design, which w^g devel- 
oped at the beginning of the project. Two of the Original 10 
learning areas were scheduled for assessment each Vear; reas- 
sessments were scheduled for about every 5 years. With the 
project now reduced to five learning areas, one area is sched- 
uled for assessment each year after 1975. Many p'^oject acti- 
vities are repeated for each learning area and for each cycle 
in accordance with the established design and employ similar 
procedures and technology. According to National Center for 
Education Statistics and project officials, measurir^g changes 
in educational performance requires such continuity, 

A single learning area assessment cycle-— from test objec- 
tives development or redevelopment to completion of the basic 
technical reporting of the data — requires about 5 Years: 

2-1/2 years for test objective and question develop- 
ment, 

1-1/2 years for the preparation and performance of 
data collection, and 

— 1 year for preliminary analysis and basic reporting. 

National Assessment officials told us they are trying to shor-. 
ten this time frame. 

< 
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CHAPTER 3 



IMPACT OF NATIONAL ASSESSMENT 

METaODSr MATERIALS, AND ASSISTANCE 

The National Assessment has contributed to American ed- 
ucation by advancing educational assessment technology and 
helping State education agencies to apply assessment techno- 
logy. The project has demonstrated the capacity to conduct 
special studies of selected learning areas. It also has pro- 
vided information on knowledge, skills, and attitudes of se- 
lected age groups of Americans in various learning areas 
assessed nationally and broken down by region, type of com- 
munity, parental education, race, and sex. (Limitations in 
the use of this data are discussed in ch. 4.) 

The National Assessment could increase its contribution 
by (1) continuing to develop and refine objective-referenced 
tests, (2) expar^Ung its assessment technology assistance to 
local educatioi gencies, and (3) providing its special stud- 
ies to all Federal agencies and others needing and willing to 
finance them. 

CONTRIBUTIONS TO ASSESSMENT TECHNOLOGY 

The National Assessment has contributed to educational 
assessment technology by developing objective-referenced 
tests and implementing a model for assessment based on these 
tests. 

Developing objective-referenced tests 

The National Assessment, together with its advisors, de- 
cided early in its planning stages not to use standardized, 
norm-referenced achievement tests; instead it developed and 
used objective-referenced tests designed to measure what 
groups of people know and can do. While standardized tests 
might have provided the means to make some interesting local. 
State, and national comparisons, they were not designed to 
provide information for determining what children have 
learned in relation to specific educational goals or objec- 
tives . 

The following differences between the makeup and use of 
National Assessment objective-referenced tests and the typi- 
cal standardized norm-referenced achievement tests were cited 
in a National Assessment report and in an article by a former 
project official. 
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—National Assessment questions measure how well stu- 
dents as a group achieve desirable educational ob- 
jectives; standardized tests compare each student with 
the average (norm) performance of other students. 

—Various National Assessment questions are geared for 
the high, average, and low ability students; stand- 
ardized test questions are aimed at the "averaae" 
child. ^ 

—National Assessment total scores reflect the number of 

students who got the correct answer on a particular 

question, and individuals do not receive scores; total 

scores on standardized tests reflect the number of 
^ correct answers a student gives. 

—National Assessment results are reported on a question- 
by-question basis; standardized test results are re- 
ported in relation to a norm group. 

—About half of the National Assessment questions used 
are made public to accompany the results, while the 
other half are kept confidential to assess performance 
changes over time; standardized achievement test ques- 
tions are rarely, if ever, made public. 

The National Assessment's method of identifying and se- 
lecting learning area objectives or revising those used in a 
previous assessment and developing questions is complex. Re- 
vising or developing objectives takes about 9 to 12 months 
and developing questions takes another 18 months. The fol- 
lowing steps occur during the development of objectives and 
questions: 

—Two series of conferences of subject matter special- 
ists, educators, and concerned citizens develop or re- 
vise objectives. ^ 

—Based on the objectives selected, the project proposes 
specifications for a pool of questions to be developed 
and supervises the work of teams of experienced sub- 
ject matter experts who make up the questions. 

—The project staff edits the questions, which are tried 
out on participant samples throughout the country. 

—Panels of subject matter specialists and educators re- 
view questions accompanied with tryout results and 
question analysis data and accept, revise, or reject 
them; the tryout and review processes are repeated for 
revised questions. 

20 
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—A panel of laymen review the accepted questions for 
relevance and for lack of bias and of f ensiveness 

—Subject matter specialists, educators, and project 
staff select questions from the pool of approved ques- 
tions to compose a set of assessment materials for 
measuring the objectives in a balanced manner. 

—A committee of the Education Commission of the State.? 
reviews the selected questions. ^taces 

Most of the educational researchers we interviewed 
stated that the National Assessment's development of 
objective-referenced tests has been a significant contribu- 
tion to assessment technology. They believe that these tests 
are more useful than standardized tests (1) in assessina 
whether educational objectives are being met and (!) JS? use 
at the local and classroom levels. Officials in several 
State education agencies we visited also complimented the Na- 
tional Assessment's work in this^ area. «"tea cne Na 

Thirty-six of the 47 States repondinq to our question- 
?nH?L^^f .^^r planning sta?ewideassessment program^ 

indicated that they used tests based on educational obiec- 
^^It's^2^^="'' P^^gra^s. AUhough it is difficuU to verify 
that States' increased use of these tests relates directly to 
the project, some educators believe that such a direct rela- 
tionship exists, our questionnaire results shSw t^at 32 
caMnn f ^ Projected 104 of the more than 8,900 local edu- 
cation agencies represented by our sample suggested imorove- 
ments m National Assessment studies. Of thlllriS States 
and 53 percent of the local agencies see a neld for fjr^her 
development of tests based on educational objectives 

Developing a model for 
objecti ve-referenced assessment 

The National Assessment has made a valuable contribution 
to educational assessment technology by developing and imni^ 
menting a model for objective-referenced assessment t^^ 

National Assessment approach to glJherinq ana- 
lyzing, and reporting data. In addition to dIvelopi'Sq' 

evaluIto^^nf^jK^^"'-^'^"''^^'°"^^ researchers and independent 
M^no ^ project cited National Assessment contribu- 

^^:":od^^r^?L^^ud?^g'^^'"°'°^%7°^^^"^ aspe^c°?s^^o?" 
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— improving the art of sampling; 

— standardizing test administration; 

— improving quality control over data gathering; 

using innovative techniques in statistically analyzin 
data on changes over time; 

pioneering new technical approaches to question and 
test development and developing unconventional ques- 
tions; and 

developing the technical feasibility for addressing 
questions relevant to policy issues, although not ful- 
filling this potential. (See ch. 4.) 

The National Assessment has facilitated the use or adap- 
tation of the model, or selected aspects of it, by state and 
local education agencies. For example, one State made an ex- 
tensive application of the model, carefully duplicating it sc 
that comparable State data was collected in each of National 
Assessment's subject matter areas. The State accepted the 
National Assessment's educational objectives as reflecting 
the State's objectives. 

ASSISTANCE TO AND IMPACT ON STATE 
AND LOCAL EDUCATION AGENCIES 

A major goal of the National Assessmeret ..project has 
f^llu^^of^""^^^^^^^ °^ project method's-' and materials 

at the State and local levels. Through technical assistance 
the project has facilitated the use of its concepts, methods, 
and materials at the State level. More emphasis, however, 
needs to be given to assisting local education agencies. 

During the past few years, interest in educational as- 
sessment at the state and local levels has increased. New 
laws have been enacted in several States mandating their own 
statewide assessment evaluation or accountability programs. 
Several States have modified their laws to require greater 

on the type of testing used by the National Assess- 
ment, that IS, testing based on objectives. According to 
project officials, the National Assessment has felt a direct 
impact of this growing movement. For example: 

—The project provided 32 onsite consultations to States 
and responded to several hundred requests for techni- 
cal assistance from State education agency personnel 
during 1974 and 1975. ^ ^ "nnex 
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— National Assessment staff contacted nearly 90 percent 
of the State officials responsible for planning and 
administering State-level assessment programs / through 
staff visits or at national conferences and other 
gatherings. 

— For 5 years the project has sponsored annual confer- 
ences on assessment and State assessment workshops for 
State and local officials. The nearly 500 attendees 
have represented 46 States, Puerto Rico, Guam, the 
Virgin Islands, and Canada. 

Assistance to States 

Our questionnaire results show that 40 States have a 
statewide assessment program and 7 others are planning one. 
Questionnaire data confirms the National Assessment's efforts 
to work with States. Thirty-one of the State respondents 
have participated in the project's workshops on assessment 
and 37 of the 48 States responding to this question have had 
discussions with colleagues who have attended from the State. 

One reason for our questionnaire survey was to obtain 
views from State education agencies about the use of National 
Assessment methods and materials for their assessment pro- 
grams. All 50 State respondents indicated that they were 
familiar with the National Assessment, but only 37 said they 
had reviewed National Assessment material sufficiently to de- 
termine its utility to their assessment programs. Of those 
37 State respondents 

32 answered yes, 3 answered no, and 2 did not respond 
when asked if they had used or planned to use the 
project's concepts, materials, or methods and 

20 indicated that National Assessment concepts, meth- 
ods, and materials were highly or very highly useful 
to their programs, 9 indicated moderate usefulness, 
7 indicated limited usefulness, and 1 did not respond. 

State use of project concepts, methods, or materials is con- 
centrated in the areas of State assessment planning and eval- 
uation. State assessment comparisons, and the development of 
educational object ives . 

When asked which organizations have provided or are 
scheduled to provide formal technical assistance for the 
State assessment program, state respondents cited consultants 
and the National Assessment most frequently. Twenty-eight of 
the 46 States responding to this question cited consultants. 
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ceived ti^hni.^? National Assessment, of those who had re- 
Jn=!! i^?'^ assistance from the project, 18 of 20 who 
answered this question stated that substantial or ver^ Jreat 

is dlvlloVarternfM f'""'^' educational commS^!?^ 

test^ ?n fi^^? ^i""^^ to standardized norm-referenced 

^" 37 out of 48 states responding indicated 

irf^?^^-^'^"''^^'^"^^ community needs to sSbstan£ialli or verv 
greatly increase its efforts to develop such alternatives ^ 

The National Assessment gives States and localitie.; hh*> 
opportunity to "piggyback" on^he project. Using this tech! 
paXd%ta?e%a°n?"''' '-tatewile assessment on 'an ex! 

ShS?e or n^r^^ ; "^"^ the National Assessment model in 
ho i? ^" 3 local school district can choos 

expandita%^.°? ^'^^^^ ^'^'^ °^ ^^e National Assessme^Jb 

NaMnn^?^A """P^^ testing and using state or 

National Assessment questions in whole or in part? 

Two States are now providing opportunities for local 
school districts to piggyback on State-level programs wJich 
already piggyback on the National Assessment. This will oro 
vide comparisons of school district results on individual 
questions with national, regional, and Sta?e results ?iese 
innovations promise relief from the present test iirburden 
^k'"?.^^^^'^^^"^^^ ""^"y schools across the country and 
t^onal 'state'^^'i^'"'?^'^^'^ cooper a^LHSong na^ 

and ^ater1fi^^-f^LL^\lTt'rmr^^^^^ 

Assessment comparisons," at least 22 States we r I usina the 
of°Jhe' 97r;f ^°h"oof°' "within-Statl" com^arfson^ 

from the NaMoni? ? ^^^"^ ' ^^^^^^ ^ad drawn questions 

oped ir^ollir.^^^^^^^^^^ -vel- 

"eSucfti^n.r^'^ i""^ assessment findings to document 
f^ndf fSrLr^^lc^a? Id-^^Lna^p-o-r^-^ ^HVH- 
Z?lT.\.Tols'.'^^' °" ^^^--^ ----- obje^\%?e% 
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— Another State, after its first statewide assessment 
in reading, was able to compare its statewide results 
with regional and national levels of per formance pro- 
vided by the National Assessment. This information 
was used to plan for more effective use of State edu- 
cation resources . One outcome of that evaluation and 
planning was a new emphasis on urban reading programs. 

To help States develop greater expertise, the National 
Assessment has also helped six States to form a new organiza- 
tion — the National Council for the Advancement of Educational 
Assessment — which will explore mutual assessment problems. 

Assistance to local school distric ts 

Based on the local education agencies represented by our 
questionnaire sample, of those responding to the .Question on 
whether they were familiar with the project, we estimated 
that 

— 54 percent had little or no information on the project 

— 30 percent were familiar only with project objectives 
or methods, 

— 14 percent had read some (11 percent) or a substantial 
number (3 percent) of project reports and critiques, 
and 

— only 2 percent had reviewed project material suffi- 
ciently to determine its usefulness to their assess- 
ment programs. 

Only about one-fourth of those able to determine its useful- 
ness consider the project's concepts, methods, and materials 
highly or very highly useful; another one-half consider them 
moderately useful; and the remainder consider them of limited 
or of no usefulness. 

According to National Assessment officials, the project 
has restricted its services to local education agencies be- 
cause of its limited resources. Usually, the project only 
provides local agencies with materials and communicates by 
telephone or mail. about general problems. In certain in- 
stances, the project provides further technical assistance if 
the local assessment is part of a State assessment plan adapt 
ing National Assessment methodology, or if the National As- 
sessment foresees the possibility of demonstrating an innova- 
tive local use of the project's procedures. 
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Perhaps because of its recent publicity campaign, the 
project received about 1,200 requests for information on 
project findings or methods from local school districts be- 
tween October and December 1975, according to project offi- 
cials. The officials also indicated that many districts 
have adopted project materials. 

The project has directly assisted at least 12 school 
districts which either participate in a State assessment pro-, 
gram or have demonstrated the staff and funding capability 
for carrying out an effective local program. 

Based on our questionnaire data, the number of local as- 
sessment programs appears to be growing, of the local educa- 
tion agencies represented by our sample that responded to the 
question, 15 percent have an assessment program and 30 per- 
cent are planning one. Only 8 percent, however, use or in- 
tend to use tests based on educational objectives, compared 
to about three-fourths of the States. 

Sources of technical assistance most frequently cited by 
local education agencies as those they have used or will use 
in order of frequency were: consultants. State education 
agencies, commercial testing services, the Office of Educa- 
tion, colleges and universities, and the National Assessment. 
Our data also shows that the larger the district is, the 
more familiar it is with the National Assessment. 

The National Assessment provided the following examples 
of how local school programs can use project assistance. 

— One suburban school district in the East was inter- 
ested in comparing the writing skills of its 13- and 
17-year-olds to other suburban children. By using 
National Assessment materials and assistance, the 
district was able to carry out a districtwide mini- 
assessment of writing which revealed that its students 
were performing above suburban levels in all but a few 
instances . 

— A school district in the Midwest wanted to take a 
fresh, hard look at the education needs of its stu- 
dents. After deciding what they should be teaching, 
the local officials looked to the National Assessment 
for help in finding out how well they were achieving 
their educational goals. They found that a number of 
the project's questions reflected local district ob- 
jectives. By selecting those questions to use in 
their local evaluation program, school officials were 
able to compare local student performance with na- 
tional performance levels. Local officials noted that 
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duplicating the National Assessment's questions and 
methods saved time and money. The evaluation program 
showed that their students' overall achievement is 
generally superior or equal to national levels. 

INFORMATION ON ACHIEVEMENTS 
IN ASSESSED LEARNING AREAS 

The National Assessment's goals include (1) acquiring 
data on the educational attainments of young Americans and 
(2) detecting changes in those attainments. 

The first assessments , which began in the spr inq of 
1969, covered citizenship, science, and writing.* The follow- 
ing table illustrates the subject areas assessed or to be as- 
sessed through 1976. 



Date Subject areas 



Mar . 


1969 


- Feb. 


1970 


Science, writing, citizenship 


Oct. 


1970 


- Aug . 


1971 


Reading, literature 


Oct. 


1971 


- Aug . 


1972 


Music, social studies 


Oct. 


1972 


- Aug . 


1973 


Mathematics, science 


Oct. 


1973 


- Aug . 


1974 


Writing, career and occupa- 










tional development 


Oct. 


1974 


- Aug . 


1975 


Reading, art 


Oct. 


1975 


- Aug . 


1976 


Social studies/citizenship 



As of January 1976, the National Assessment had pub- 
lished 51 reports based on asri^essment results for selected 
age groups in 10 learning areas. The reports included the 
following . 

— Political Knowledge and Attitudes drew information 
from the social studies assessment results that could 
possibly contribute information regarding America's 
political health. 

— Contemporary Social Issues describes America's social 
awareness . 

— A Perspective on the First Music Assessment provides 
National Assessment results in the context of expert 
consultant views on research, curriculum, and teaching 
issues related to music . 

— Consumer Math, Selected Results from the First Na - 
tional Assessment of Mathematics focuses upon mathema- 
tical skills needed by the American consumer to func- 
tion effectively in the marketplace. 
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National achieveinent patterns 



Examples of the information the National Assessment has 
developed for several assessed subject areas follow. 

— The 1969-70 science assessment showed that males' over- 
all performance was higher than females' at all ages 
and that the gap widened with age. Males demonstrated 
a more thorough knowledge of physical science, and fe- 
males seemed to have a better knowledge of biological 
science . 

— The 1970-71 reading assessment showed that most Ameri- 
cans tested read well enough to accomplish simple prac- 
tical kinds of tasks. The overall reading ability of 
blacks was lower than any other group s,=»'^.f \ , includ- 
ing geographic and other groupings. Scaoox-age males 
read less well than school-age females, but adult men 
and women had about the same reading ability. People 
from families in which neither parent had gone to high 
school and those from inner city areas read less well 
than most other groups of people. 

— The 1971-72 social studies assessment shewed that 
young Americans lacked knowledge of the fundamentals 
of politics and civil rights. 

— The 1972-73 science assessment showed a decline from 
the 1969-70 assessment at all three school ages sur- 
veyed in most questions measuring knowledge and skills. 
Two groups that achieved lower scores on the earlier 
test, students in the Southeast region and rural resi- 
dents, improved their relative national standing but 
still performed generally below the national level. 

— The 1973-74 writing assessmer.t showed a decline from 
the results of the 1969-70 assessment in the writing 
ability of 13- and 17-year-olds. Nine-year-olds 
gained slightly in writing ability over this period. 

SPECIAL STUDIES OF 
SELECTED LEARNING AREA S 

A National Assessment goal is to conduct special studies 
m selectea areas of educational attainment. The National 
Assessment's system for sampling national populations allows 
special studies— intended to answer some specific questions— 
to bo made m conjunction with regular assessments without 
overburdening data collection efforts. This provides addi- 
tional data while allowing time and resource savings. 
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The National Assessment's major effort in this area has 
been to assist the Office of Education's Right to Read pro- 
gram by making a special study of functional literacy to meet 
the program's needs. Right to Read is intended to reduce na- 
tional illiteracy. The study was intended to discover how 
many 17-year-olds are unable to read well enough for produc- 
tive employment and citizenship. This assessment involved 
administering two extra test booklets of specially selected 
National Assessment reading exercises to 5,200 17-year-olds 
for 2 successive years. The cost of about $350,000 was paid 
by Right to Read. A Right to Read program official told us 
that this study has been useful in defining program needs 
and suggesting areas needing curriculum modification to 
school administrators and teachers. 

Although the National Assessment is planning to conduct 
other special studies in the next few years as part of its 
contractual obligations to HEW, Right to Read is the only 
Federal program that has separately reimbursed the project 
for a special study. 

Another speci-al study, planned for fiscal year 1976, is 
in the area of basic mathematics. Tha development of the as- 
sessment is based on previous National Assessment work in 
mathematics. Questions selected were used in the 1972-73 
mathematics assessment . 

Also being studied for possible later assessment is an 
"index of basic skills" — to determine what skill^r are funda- 
mental to active and productive participation in American 
society and what questions might best detect the presence or 
absence of those skills. 

According to National Assessment and National Center for 
Education Statistics officials, the special study approach — 
exemplified by the assessment made for the Right to Read pro- 
gram — is worthwhile, and efforts of this nature are an effi- 
cient and timely use of the National Assessment's capabili- 
ties. The project's report on its goals and accomplishments 
from 1969 through 1975 states that it would consider con- 
ducting a special study whenever an apparent need exists and 
sufficient staff time and resources are available. The Na- 
tional Assessment's 5-year plan for fiscal years 1976-80 
also states that it intends to cooperate with others needing 
such data who are willing to bear the expense. However, Na- 
tional Assessment officials acknowledge that they have not 
adequately informed other agencies about the possibility of 
meeting their needs through this approach. 
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In view of the more than $18 billion outlay for educa- 
tion programs projected in the President's fiscal year 1977 
budget, opportunities likely exist for other Federal agencies 
to benefit from the special study approach in defining and 
meeting their program needs. 

CONCLUSIONS 

The National Assessment has contributed to American ed- 
ucation f but cer tain improvements are war ranted. The Na- 
tional Assessment has contributed to assessment technology 
by developing objective-referenced tests and implementing a 
model for assessment based on objectives. The project should 
cont inue developing and refining these tests to increase 
their acceptance and use by the educational community. The 
project has also facilitated, through technical assistance, 
the implementation of its methods and materials at the State 
level. However, when possible within the available funding, 
more emphasis should be given to similarly assisting local 
educational agencies. Finally, the National Assessment has 
demonstrated its capability for conducting special studies. 
Project officials agree that they need to better inform po- 
tential users about the potential of this approach. Oppor- 
tunities likely exist for other Federal agencies to benefit 
from this approach in defining and meeting their program 
needs. 

RECOMMENDATIONS TO THE SECRETARY OF HEW 

To better utilize National Assessment capabilities, we 
recommend that the Secretary urge project management to : 

— Continue developing and refining objective-referenced 
tests • 

— Increase its technical assistance to local education 
agencies regarding its concepts, materials, and meth- 
ods when possible within the available funding. 

— Increase its efforts to serve Federal agencies and 
others by providing special studies for those who need 
and are willing to finance them. To help accomplish 
this, the project should better inform potential users 
of its capabilities in this area. 

AGENCY COMMENTS 

In a June 3, 1976, letter (see app. Ill ) , HEW agreed 
ith the above recommendations and described the following 
actions planned to implement them. 
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— Having the National Assessment continue developing 
and refining objective-referenced tests. 

—Urging the project to attempt, within the available 
funding, to increase its technical assistance to local 
education agencies. 

—Encouraging the National Assessment to increase its 
efforts to provide special studies for various agen- 
cies and organizations. The National Center for Edu- 
cation Statistics and the project will also expand 
their efforts to inform Government agencies about the 
project's methods, materials, and findings and at- 
tempt to address the specific data needs of the var- 
ious agencies. 

THE NATIONAL ASSESSMENT'S COMMENTS 

In an April 27, 1976, letter (see app. IV), the National 
Assessment agreed with our recommendations and described the 
following actions being taken or planned to implement them. 

— Attempting to refine objective-referenced testing 
technology by insuring more representative coverage 
for science and mathematics assessments and con- 
ducting feasibility studies to solve problems of 
how to measure such difficult learning areas as 
problem-solving or speaking and listening skills. 

Creating more definitive documents to make it easier 
for local education agencies to use the project's 
materials . 

— Exploring assistance that could be given to State 
agencies to enable them to provide needed technical 
assistance to local education agencies. 

—More adequately informing those who may use National 
Assessment special studies of the project's capabili- 
ties in this area and expanding the project's special 
study efforts. 
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CHAPTER 4 
IMPROVEMENTS NEEDED TO INCREASE THE 
USEFULNESS OF NATIONAL ASSESSMENT DATA 



National Assessment data has been of limited usefulness 
to educational decisionmakers and, therefore, the project 
has been prevented from achieving its basic mission. To in- 
crease the usefulness of its data, the project needs to 

— improve its planning for and attention to the needs 
of educational policymakers , 

— improve data interpretation, 

— provide for comparison of project test results with 
performance standards, 

— revise its assessment approach so that it provides 
data appropriate for decisionmakers' needs, and 

— improve dissemination of results. 

National, State, and local officials have also made 
suggestions for increasing data usefulness, such as including 
finer geographic breakdowns and more demographic and back- 
ground variables on assessment participants. 

LIMITED USE OF NATIONAL 
ASSESSMENT DATA 

The National Assessment's April 1974 5-year plan and 
other reports state that the project's basic mission is to be 
an important source of information useful to educational de- 
cisionmakers and practitioners in identifying problems, set- 
ting priorities, and determining progress. The National As- 
sessment's report on its goals and accomplishments from 1969 
through 1975 indicates that the project seeks to provide edu- 
cational decisionmakers at the national level with concrete 
information to help them set priorities and allocate re- 
sources. The National Assessment's 5-year plan states that 
other data users include 

— State and local education officials, 

— curriculum developers and researchers, and 

— the public. 
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Although all the officials we interviewed at HEW, re- 
search organizations, and test publishing firms were aware 
of the National Assessment, very few indicated that they had 
used project data and few could cite any use for the data as 
it is currently collected and presented. 

Most congressional staff members and HEW officials (in- 
cluding those in the Office of the Assistant Secretary for 
Education and the Office of the Assistant Secretary for Plan- 
ning and Evaluation) that we spoke with said that National 
Assessment data has not been used to set priorities or 
stimulate either legislative or program changes. 

National Assessment officials, some officials from HEW 
(including the offices of the assistant secretaries), and 
some congressional staff members told us they believed Na- 
tional Assessment data was worth collecting generally be- 
cause it may become very useful as more and more trend data 
is accumulated. They contended that over time, as more data 
is accumulated, more opportunities will arise for the data 
to indicate changes needed in legislation, programs, and cur- 
r iculums . 

State and local respondents to our questionnaire indi- 
cated that they have not used and do not plan to use -National 
Assessment data for resource allocation purposes. Most of 
the local education agencies represented by our sample re- 
sponses indicated that they have little or no information on 
the National Assessment. Only 2 percent of the local educa- 
tion agencies represented by our sample responses and about 
75 percent of the State respondents said they were suffi- 
ciently familiar with National Assessment material to be able 
to determine its usefulness to their assessment programs. 

The National Assessment's report on its goals and accom- 
plishments from 1969 through 1975 states that its data has 
noc stimulated expected curriculum redesign efforts. Accord- 
ing to a major test publisher we spoke with, project results 
were not very relevant to curriculum objectives and were too 
far removed from the classroom level to be of use. HEW and 
project officials disagreed with this, stating that available 
evidence, including a series of articles in two professional 
journals for teachers discussing the implications of assess- 
ment results for curriculum and instruction, seems to contra- 
dict this opinion. (See pp. 58 and 64.) HEW and project 
officials noted, however, that the project clearly needs to 
better inform edurr^ional practitioners — such as teachers 
and curriculum sj, alists — about the project and its find- 
ings. 
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Three of the four test publishers interviewed stated 
that they could not see any use for the project's data in 
their testing activities. One thought that the data was use- 
ful as general information for tha public, but only because 
it presents a broad view. 

According to officials we interviewed at two State edu-* 
cation agencies, the project's data is often not useful to 
them because the objectives on which the tests are based are 
different from the States' curriculum objectives. Officials 
from a university-affiliated research organization said that 
the National Assessment data must be tied to classroom curri- 
culum objectives or it will remain unused. They consider the 
classroom teacher as potentially the greatest user. 

Limited research application 

Because the National Institute of Education is the re- 
search arm of HEW' s Education Division, project officials ex- 
pected that their data would stimulate Institute substudies 
or research to determine causes for particular sets of data. 
National Assessment officials noted, however, that no such 
studies have^jbeen made. ' Institute officials agreed and ex- 
plained that the National Assessment's objectives and pur- 
poses for data collection- differ from the Institute's and 
that this limits the application of Institute research ques- 
tions to the National Assessment's data. 

Other researchers stated that the project's data has not 
stimulated them to explore the data further. The project's 
report on its goals and accomplishments from 1969 through 
1975 characterized the response of researchers, professional 
groups, legislatures, and others to its data as generally 
disappointing. The Director of the National Assessment told 
us that the project needs to get more educational rese^chers 
to work with proje^ct data and plans to take steps to bring 
this about . 

BETTER PLANNING'^NEEDED TO ^ 
MEET THE NEEDS OF DECISIONMAKERS 

In our opinion, for survey data of the type the National 
Assessment collects to be most useful for decisionmaking, it 
is necersary to have adequate planning which includes obtain- 
ing input from those who will or may use the data. Such 
planning should increase the likelihood that (1) the data 
needs and decisions to be made are adequately defined, (2) 
the data will be usable, and (3) the data will be effec- 
tively communicated to the intended decisionmakers. 
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Not only should the Congress, HEW officials, res. q.ch- 
ers, and other educational decisionmakers know what . • 
project has produced and how the data can be used, b' la- 
tional Assessment officials should know on a continuinr 
basis what short- and long-term problems or questions these 
officials have that the project can try to answer. 

Many national. State, and local decisionmakers indicated 
that National Assessment data has not been especially mean- 
ingful or relevant to them. Congressional staff said the 
data has been of little legislative use. HEW officials and 
congressional staff told us that National Assessment results 
often have not been relevant to policy questions because of 
the form, generality, and lack of connection between the data 
collected and decisionmakers' needs. Reports have been too 
technical, too lengthy, and not keyed to policy questions. 

Congressional staff and HEW and State education agency 
officials told us that the National Assessment has made only 
limited efforts to link its data collecting efforts to deci- 
sionmakers~the Congress, HEW, State education agencies, and 
others. 

One educational researcher also told us that project re- 
sults have not been relevant to policy questions and sug- 
gested that the project set up a panel of experts to help it 
determine, 2 to 4 years in advance, future policy questions 
that will need assessment data. In his opinion, the project 
has developed the technical capability to address such policy 
questions. 

An evaluation of the National Assessment made for the 
National Center for Education Statistics also concluded that 
the project staff and the Education Commission of the States 
should attempt to open new meaningful contacts with decision- 
makers to discover their policy concerns and data needs. The 
evaluation noted that, unless the data is more relevant to 
policy questions and suggests possible causes of deficien- 
cies, it is not reasonable to expect wide use of the data in 
social and educational decisionmaking. 

The Director of the National Assessment told us that he 
wants to become more responsive to decisionmakers' needs. He 
stated that meetings have been initiated with national deci- 
sionmakers and that regional conferences with State and local 
officials are being planned to get their views on changes 
needed in the National Assessment. 
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NEED TO IMPROVE DATA INTERPRETATION 



The usefulness of the National Assessment data depends 
greatly on its interpretation. Until the completion of our 
fieldwork in October 1975, the National Assessment had con- 
sistently refrained from interpreting its own findings. The 
project described questions asked and answers given without 
discussing the implications for the educational community, 
making inferences, or providing hints as to possible causes 
of deficiencies. This policy enabled the project (1) to 
avoid the possible political repercussions of appearing to 
be a national agency attempting to guide or dictate local 
curriculums and (2) to maintain an objective position in re- 
lation to the reader or user. 

The National Assessment's policy, according to its offi- 
cials, had been to gather data and encourage interpretation 
by those who had the expertise in the subject matter and who 
could apply criteria to the findings in order to discuss cur- 
ricular implications. However, subject matter specialists 
and other outside groups did not use the data just because it 
was available as the project had expected, according to the 
project's report on its goals and accomplishments from 1969 
through 1975. 

Although disappointed by the few efforts made by the 
educational community to interpret its data, the project con- 
tinued to feel that it should not interpret its own data. 
Therefore, in 1973 the project began to contract with subject 
matter specialists and organizations for interpretation. The 
National Assessment does not endorse or take responsibility 
for the data interpretations of these professional groups. 
Through fiscal year 1975 about $150,000 had been spent for 
interpretative studies by organizations representing teachers 
of mathematics, English, science, and social studies. In our 
view, this is a minimal effort in relation to the millions of 
dollars spent for data collection. 

According to a nationally recognized expert on testing 
and measurement, a major problem with the interpretations 
that have been made is that they have refrained from going 
beyond highly precise and highly probable statements , and 
policy decisions do not always reauire such precision and 
certainty. An HEW official agreed, stating that what is of- 
ten required is trend data and an indication of the data's 
rel iabil ity . 

Most State officials, researchers, test publishers, cur- 
riculum developers, and Office of Education officials we in- 
terviewed recommended that, to increase data utilization, the 
National Assessment interpret its own data. In our opinion. 
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the National Assessment can best interpret its own data, 
since it is most familiar with that data. 

The Director of the National Assessment agreed that the 
project needs to interpret its data and that the lack of ade- 
quate interpretation is a valid criticism. He saia that the 
National Assessment is starting to interpret its data by pre- 
paring short summary interpretive reports. We believe that 
these initial efforts should be continued and increased. 

NEED TO COMPARE TEST RESULTS 
W ITH PERFORMANCE STANDAR DS 

A factor contributing to the lack of interpretation of 
National Assessment data is the lack of standards against 
which test data can be compared to judge performance. An 
example of a performance standard might be; "80 percent of 
^the 9-year-olds should meet this objective 90 percent of the 
"time." The National Assessment contends that no one knows 
for sure what a reasonable percentage of success should be, 
partly because concrete achievement data has never been avail- 
able. Because the project has not attempted to define them, 
estaolishing performance levels is left to the reader or in- 
terpreter of National Assessment reports. 

Officials of two States told us that the people who work 
with the data should interpret it and draw implications in 
relation to standards. Three prominent researchers told us 
that the National Assessment should set standards for com- 
parison to allow meaningful use and interpretation of project 
results. One researcher stated that he considers the pro- 
ject's current practices to be an invitation for misinterpre- 
tation. In our opinion, unless meaningful performance com- 
parisons can be made. States, localities, and other data 
users are not as likely to find the National Assessment data 
to be useful . 

An HEW official said that the lack of performance stand- 
ards limits interpretive efforts. National Assessment data 
does not meaningfully assess how well groups of students are 
doing in relation to what they should be capable of doing. 
According to one State official from a State which has mod- 
eled its assessment after the National Assessment's model, 
some local school districts are setting standards for their 
local assessment to compare with State and National Assess- 
ment data. 

Tne National Assessment recognizes that the lack of 
standards prevents (1) judgments on whether the performance 
of various groups is satisfactory and (2) cross-comparisons 
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between subject areas (for example, how good or bad perfor- 
mance in reading compares to performance in science). The 
National Assessment's 5-year plan for fiscal years 1976-80 
states that the project is exploring this Question and is 
planning, over the next several years, to develop procedures 
for comparing National Assessment results to meaningful per- 
formance standards . 

NEED TO REVISE THE 
ASSESSMENT APPROACH 

Assessment results are more useful when the assessment 
is designed so that the approach is related to the intended 
use of the data and the needs of the intended users. The 
project's assessment approach includes the subject. or learn- 
ing areas assessed; the test questions and objectives, sample 
size, and geographic and demographic variables used; and the 
methods of reporting and disseminating results. 

With the limited funding available, the National Assess- 
ment believes the best approach to collecting useful informa- 
tion is documenting educational deficiencies and inequities 
and monitoring changes in them to attempt to influence policy. 
The issue, however, is whether the approach being followed is 
too narrowly conceived and inflexible, limiting the data's 
usefulness and preventing the project from undertaking pos- 
sibly more valuable functions. 

According to State officials, representatives of private 
and Government-sponsored research organizations, test publish- 
ers, HEW officials, and congressional staff, a major problem 
which limits data interpretations by either National Assess- 
ment staff, consultants to the project, or users results from 
the project's assessment approach. 

HEW officials not directly responsible for the project 
specified that the data is too general to be of programmatic 
use, contains too few variables on background and demographic 
factors, is based on a sample unnecessarily large for issues 
of concern to decisionmakers, and needs too long a lead time. 
These views were shared by state decisionmakers, researchers, 
and test publishers. 

The National Center for Education Statistics-sponsored 
evaluation of National Assessment previously mentioned noted 
that the substance of National Assessment data — attainments 
in traditional subject areas and changes over time — is of 
greatest interest at the State and local levels but the form 
in which the data is collected and reported — for example, 
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only on regional and national bases — greatly limits data use- 
fulness to them. On the other hand, the substance is of min- 
imal interest at the national level, where the form is most 
applicable. Thus the National Assessment data is not satis- 
fying Federal, State, or local levels. 

The evaluators suggested that the solution to this di- 
lemma is to change either the substance, the form, or both. 
Because the project's chief current contribution to State 
and local levels seems to be in helping them to use aspects 
of its assessment model, the evaluators specifically sug- 
gested that the National Assessment find out what kind of 
information is needed at the national level and provide it. 
They cautioned against the project being too tied to Its 
present assessment approach, stating that, if data useful- 
ness cannot be increased through this approach, the neces- 
sary changes should be made to make the project a responsive 
instrument in solving education problems. 

In our view the National Assessment has not demonstrated 
that the benefits of its assessment approach warrant the 
costs. The project's approach needs to be thoroughly reeval- 
uated and redirected to' improve the usefulness of its re- 
sults. This includes exploring the costs and benefits of al- 
ternative approaches . 

Project officials partially attribute the problems in 
the project's assessment approach to the historical and so- 
cial environment existing when the National Assessment was 
conceived and implemented. 

— The originators of the National Assessment wanted a 
systematic effort designed to gather general infor- 
mation about educational outcomes in the United States. 
The National Assessment was not designed to answer 
specific educational questions. 

— The National Assessment was conceived as only one 
information-gathering project, which, along with 
others, would help to serve the needs of educational 
decisionmakers . 

Therefore, political and other reasons have precluded the 
project since its inception from providing State- and local- 
level data. Nevertheless, the originators thought that as- 
sessment data would help national and State legislators in 
their funding decisions, as well as local school board mem- 
bers, teachers, and program administrators. 
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NEED TO IMPROVE DISSEMINATION 



Effective dissemination of results increases their use- 
fulness. The dissemination strategy should be an integral 
part of the project planning process. 

In the early years the National Assessment staff was 
largely concerned with the problems of contract monitoring, 
sampling, data analysis, and test objectives and question 
development. Little time, money, or staff were applied to 
developing reports. In fact, the staff did not fully assume 
responsibility for communicating its results until fiscal 
year 1971, and a major shift of attention to report writing 
and dissemination was not made until fiscal year 1973. Pro- 
ject officials attribute this shift to the need to get in- 
formation to different audiences in different ways. 

Our questionnaire results show that most State respond- 
ents are well informed about the National Assessment but that 
its impact at the local level has been minimal. Only 2 per- 
cent of the local education agencies represented by our sam- 
ple have extensively reviewed project material, and 54 per- 
cent have little or no information on the National Assessment. 
At 73 percent of the local education agencies. National As- 
sessment reports have not been read or evaluated. 

Project officials told us that they have minimal con- 
trol over the dissemination of their reports because they 
are sold through the Government Printing Office. They said 
that such dissemination has caused them some concern. First, 
the National Assessment cannot identify who data users or 
buyers are, and second, potential buyers of project reports 
might have trouble obtaining reports. Project staff based 
the latter statement on a test they made in which they re- 
quested several recent reports but received only three, all 
dating back to 1971. According to National Center for Edu- 
cation Statistics officials, the Government Printing Office 
is cooperating with them to try to solve this problem. 

National Assessment officials said that, within the con- 
straints of limited staff and resources, they are attempting 
to improve the dissemination of information to decisionmakers 
by 

— mailing information on each newly published report to 
about 200 people and mailing a bimonthly newsletter 
to about 23,000; 

— distributing press releases and holding press confer- 
ences when important findings are released; 
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— releasing previously unreleased test objectives and 
questions and providing access to computer data tapes 
of all project findings; 

— writing articles for magazines, journals, and news- 
papers; and 

— including reports in the Education Resources Informa- 
tion Center, commonly known as ERIC. 

In addition, the National Assessment has tried to im- 
prove the dissemination of assessment data by 

— simplifying and shortening reports, 

— writing different types of reports for different au- 
diences, and 

— responding to specific data analysis reauests from the 
Office of Education and others. 

OTHER SUGGESTIONS FOR 
INCREASING DATA USEFULNESS 

Many State and local education officials and others sug- 
gested that National Assessment data include finer geographic 
breakdowns (such as by State) and many more demographic and 
background variables (such as information on the family and 
educational background of tho.?a sampled). 

The evaluation of the National Assessment previously 
referred to recommended that the project consider including 
additional background variables and that it study carefully 
the usefulness of the current sampling approach, which is on 
a regional basis, to determine whether eliminating it and 
only retaining a national sample would save much money or 
time. 

Of the 32 State questionnaire respondents and the small 
number of local education agencies (104 of the more than 
8,900 represented by our sample responses) who suggested im- 
provements in National Assessment studies: 

— 15 States and 65 percent of the local education agen- 
cies recommended smaller geographic breakdowns, such 
as by State or county, 

— 15 States and 53 percent of the local education agen- 
cies saw a need for further developing tests based on 
educational objectives. 
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— 11 states and 40 percent of the local education agen- 
cies believed there is a need to relate National As- 
sessment data to Office of Education elementary and 
secondary education program evaluations. 

— Only 3 States but 36 percent of the local education 
agencies believed more demographic breakdowns are 
needed. 



— 4 States and 12 percent of the local education agen- 
cies believed an increase is needed in technical guid- 
ance to initiate comparable State assessments. 

~2 States and 13 percent of the local education agen- 
cies believed an increase is needed in the number of 
age groups assessed. 

Officials we interviewed from two States said that the 
National Assessment should move from being primarily a data- 
collecting operation to a service-oriented one, providing 
(1) models for assessment, (2) guidance or assistance, and 
(3) storage of and access to subject area objectives and 
test questions. 

We note that several of the above suggestions irwolve 
different costs and consequences. For example, providing 
data on a State-by-State basis would mean, according to Na- 
tional Assessment officials, increasing the sample size for 
each State to the level currently used for a region and in- 
creasing current costs tenfold. 

On the other hand. National Assessment officials agreed 
that restricting their work to focus on service and research 
could reduce costs. This would also mean, however, abandon- 
ing the assessment approach and the baseline data developed 
thus far. 



A concentrated effort to assist States and localities 
would involve some problems because of different assessment 
requirements of each State and locality and the costs in- 
volved. These differences could involve potential problems 
ranging from State laws requiring the use of norm-referenced 
tests for assessment, to different curriculum objectives, to 
preferences regarding collecting data on individual pupil 
performance versus groups of pupils. 

Although States originally opposed State-level data col- 
lection by the National Assessment, our Questionnaire results 
show a tendency toward greater acceptance of this approach 
National Assessment officials agreed that some States are now 
willing to let the National Assessment collect State data. 
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According to project officials, they are willing to cooperate 
with any State wanting assistance and can expand the National 
Assessment sample for any State at the State's expense. 

National Assessment officials are studying the possibil- 
ity of substituting other background variables — which are 
compatible with factors in other educational studies — for 
currently used variables which overlap. They agree that this 
could assist in presenting better descriptive data. 

CONCLUSIONS 

National Assessment results have been of limited useful- 
ness to educational decisionmakers, researchers, and practi- 
tioners. This has prevented the project from achieving its 
basic mission — to be an important source of information use- 
ful to educational decisionmakers and practitioners in iden- 
tifying problems, setting priorities, and determining progress 

Reasons for limited use of project results are 

— planning which is inadequate to recognize the needs 
of educational policymakers, 

— lack of attention to data interpretation by the project 

— lack of comparison with performance standards, 

— an assessment approach which provides inappropriate 
data for decisionmaking, and 

— inadequate dissemination of results. 

The National Assessment has demonstrated that acquiring 
data on educational attainments and measuring changes over 
time are possible. (See ch. 3.) It has not demonstrated 
that the benefits of the project's assessment approach war- 
rant the costs. The benefits and costs of alternative ap- 
proaches also need to be explored. 

RECOMMENDATIONS TO THE SECRETARY OF HEW 

TO make the project's results more useful, we recommend 
that the Secretary redirect the project by (1) identifying 
the informational and other needs of decisionmakers, (2) de- 
termining the feasibility and cost effectiveness of alterna- 
tive approaches for collecting and reporting educational as- 
sessment data and providing other services to satisfy those 
needs, and (3) deciding on the assessment approach to be used. 
Although consultation with project officials seems necessary 
during this process, we believe it is also important to seek 



43 

34 



the views of users or potential users of project results at 
national. State, and local levels and of recognized experts 
in educational management, assessment, and research. 

Examples of possible alternative strategies for the Na- 
tional Assessment include: 

— Restricting the project's data collection efforts and 
primarily providing assessment models, test objec- 
tives and questions, and technical assistance con- 
cerning assessment to States, local school districts, 
etc. 

— Expanding the project's data collection efforts to 
provide more information on geographic, demographic, 
or background variables. 

— Focusing data collection efforts on the information 
needs of one level of government, such as the Federal 
level, and restructuring subject areas, sample size, 
and demographic and background variables to serve 
those needs within a shorter tim6 frame. 

— Making more modest changes in the present mixture or 
manner of providing technical assistance, planned as- 
sessment data, and special studies. 

To increase the use and improve the usefulness of Na- 
tional Assessment data, we recommend that the Secretary re- 
quire project management to: 

— Establish continuous dialogues with the Congress, exe- 
cutive agencies, and State and local policymakers to 
determine their data needs and how the National As- 
sessment can best meet those needs. 

— Continue and increase recent efforts to interpret pro- 
ject data so as to rely less on others for data inter- 
pretation . 

— Provide for comparison of its test results with perfor- 
mance standards by giving greater emphasis to develop- 
ing the procedures needed to compare those results to 
generally acceptable non-Federal standards. 

— Improve its communication and cooperation with the Na- 
tional Institute of Education and other educational 
researchers to facilitate possible further research on 
and interpretation and use of National Assessment re- 

. suits. 



44 

35 



— Improve dissemination of National Assessment results. 

AGENCY COMMENTS 

Regarding our recommendation that it redirect the pro- 
ject, HEW stated that it does not believe redirection is war- 
ranted until an extensive reexamination of the project's 
fundamental assumptions and guiding principles has been com- 
pleted in conjunction with substantive experts, user groups, 
and others. The National Center for Education Statistics is 
conducting internal studies and consultations with various 
agencies and individuals to decide on the necessity, nature, 
or extent of any redirection. HEW stated that the results 
of this reexamination should enable agency officials to de- 
termine whether or not the project should move to a substan- 
tially new design approach or attempt to improve and expand 
the present model and conceptual framework. HEW also noted 
that in any new model, provision must be made for inter- 
relating the present and proposed data systems to conserve 
the value of the data that has been accumulated for a decade 
at the cost of approximately $40 million. 

Regarding our recommendation that HEW require the pro- 
ject to give greater emphasis to developing the procedures 
needed to compare its test results to generally acceptable 
standards, HEW stated that it will (1) explore with the 
project's staff the problem of standards, including the pro- 
cedures and technical considerations associated with achiev- 
ing comparisons with standards, and (2) encourage efforts to 
provide the technical and methodological guidance needed for 
policymakers at various levels to make their own comparisons 
or judgments concerning appropriate standards. The project 
went beyond this and agreed to provide for comparison of 
its test results with performance standards. 

We believe that lack of performance standards is an impor 
tant factor limiting the usefulness of the project's data 
and that HEW should require the project to put greater em- 
phasis on solving this problem. in spite of HEW's unwilling- 
ness to make this a requirement, the project's commitment to 
provide for such comparison with standards is a good first 
step toward solving the problem. 

HEW agreed with our other recommendations and described 
the following actions t'-^'<en or planned to implement them. 

— Focusing the project's greatest efforts during the 
next few years on establishing continuous dialogues 
with Federal, State, and local policymakers to deter- 
mine how the project can best meet their needs. This 
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includes holding a series of meetings throughout the 
country with various users and potential users of 
project data, it also includes examining existing 
practices and procedures at Federal, State, and 
local levels where assessment, research, or survey 
data is used in the decisionmaking process to identify 
(1) information needs for educational decisionmaking, 
with particular emphasis on the Federal level, and (2) 
information and other needs for curriculum development 
and improvement. This process would be intended to 
probe in depth how specific information is used for de- 
cisionmaking, identify specific data needed for se- 
lected kinds of decisionmaking, and determine the best 
ways of establishing continuous dialogues with user 
groups • 

— Requiring the project to continue and expand its data 
interpretation efforts and to strengthen its present 
analysis capability. HEW pointed out that the future 
constraint it envisioned is the project's inability 
to retain enough experts to permit it to interpret all 
its findings* Therefore, researchers and others would 
also De encouraged to continue to examine and report 
on project data from a variety of perspectives, 

— Having the National Center for Education Statistics 
(1) give more attention to both formal and informal 
relationships with the National Institute of Educa- 
tion and other Federal agencies and (2) explore var- 
ious options available to encourage further analysis 
of project data by educational researchers. 

— Continuing to seek the cooperation of the Government 
Printing Office in improving the distribution of Na- 
tional Assessment reports and exploring the possibil- 
ity of the project distributing the reports and other 
publications directly. 

THE NATIONAL A SSESSMENT'S COMMENTS 

The National Assessment agreed with all our recommenda- 
tions and described the following actions being taken or 
planned to implement them, 

— Identifying decisionmakers' information needs and 
evaluating alternative approaches for presenting 
project data, 

— Creating an Office of Application within the National 
Assessment to identify assessment-related information 
of Federal, State, and local education decisionmakers, 
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Conducting a series of eight future-focus conferences 
with teachers, local and State administrators, members 
of national educational organizations, Federal offi- 
cials, and university researchers to provide input for 
the project's planning process, 

•Arranging meetings with key congressional leaders to 
try to establish continuing dialogues with the Con- 
gress • 

-Continuing its efforts to interpret project data. 

-Attempting to develop and disseminate a methodology 
with which State and local education agencies can es- 
tablish their own standards and submitting a foundation 
grant proposal in an attempt to establish the feasi- 
bility of contrasting actual performance with prees- 
tablished standards. 

-Attempting to establish a working relationship with 
the National Institute of Education through meetings, 
briefings, and the exchange of information and taking 
other steps to establish working relationships with 
other education researchers to facilitate research on 
and use of project results. 

-Continuing to improve dissemination of project results 
through such actions as increased newsletter circula- 
tion, presentation of papers at national conferences 
and professional meetings, and media coverag<5 of pro- 
ject results. 
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RBflOLTS OF QAO'S 
STATB BDDCATIOM AQPtCT QDMTIOIIWAIIUI 



National Assessaent of Educational Progresd 

1. Hov familiar are you with the National Assess- 
nent of Educational Progress (NAB?) project 
sponsored by the Office of Education (OE) 
through the Education Conission of the States? 
(Check the one response which best expresses 
your faailiarity with NAEP) (note c) 

Have little or no inforaation 

Paniliar with the purpose and aajor ob* 
jectives 

Paniliar with the project Mthods and 

specific objectives 

Read sone project reports and critiques 

Read a substantial nuaber of project re- 
ports and critiques 

Conducted sufficient reviews and evalua- 
tions of NAEP material to det^mine its 
utility in your assessment program 



2. If you have read or evaluated any of NABP's 

survey results and/or objective booklet^, which 
of the following NAEP survey learning areas are 
you familiar with? If none, skip this question. 
(Indicate your answer by checking the appropri- 
ate responses.) 



Number 

responding 
from 50 
( note a ) 



Responses 

Percent 
Number (note b) 



50 



0 

8 

2 
8 



74 
100 
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social studies 

Note: If you have checked the last response in ques- 
tion 1 above indicating that you have assessed 
the utility of NABP applications to your 
State, continue. If not, go to question 7. 



32 
29 
36 
41 
18 
18 
24 



71 
64 
80 
91 
40 
40 
53 
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3. What utility do the concepts, methods and materi- 
als developed by NAEP have for your State educa- 
tion program? (Check the one response which best 
expresses your position.) 

Little or no utility 

A limited degree of utility 

A moderate degree of utility 

A high degree of utility 

Very high degree of utility 



4. Have you used or do you plan to use any of the 
NAEP concepts, methods, or materials in your 
State education program? 



Yes (continue) 

No (go to question 7) 



5. On which of the following State education ac- 
tivities have you either used or planned to use 
NAEP concepts, methods, or materials? 

State policy planning 

State assessment planning and evaluation 

Development of educational objectives 

Instructional methods and curriculum plan- 
ning and evaluation 

Resource allocation 

Educational accountability 

state assessment comparisons 

Within State assessment comparisons 

Program evaluations 

Diagnostic and/or individual needs as- 
sessment methodology 

Other (please specify) 



Number 
responding 
from 50 
( note a ) 



36 



35 



32 



Responses 

Percent 
Number (note b) 



0 
7 
9 
12 
8 



32 
3 



7 
23 
13 

3 
0 
10 
16 
12 
7 

4 

5 



0 
19 
25 
33 
22 
d/100 



91 

9 

100 



22 
72 
41 

9 

0 
31 
50 
38 
22 



13 
16 
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6. What wfiys would you suggest to improve the 
utility of NAEP studies to you? (Check all 
that apply* ) 

None 

wider dissevination of NAEP studies 

Technical guidance to initiate cowpaf 

able State assessments 

Smaller geographic breakdowns^ such as by 

State or county 

More detailed demographic breakdowns 

Relate NAEP data to OE-BSEA (Elementary 

and Secondary Education Act) program 
evaluations 

Increase the scope of the development of 

criterion-referenced tests (note e) 

Increase the range of age assessments 

Improve the quality of the criterion- 
referenced tests (e.g., improve validity, 
reduce biases and error, etc.) 

Other (please specify) 

. Have you attended an annual State assessment 
workshop/seminar sponsored by NAEP? 

Yes 

No 



Number 
responding 
from 50 
(note a) 



32 



50 



Have you had discussions concerning the benefits 
of this NAEP-sponsored workshop/seminar with any 
of your colleagues within the State who have at- 
tended? 

Yes 

No 



If yes to either question 7 or 8, continue; if no to 
both, skip to 10. 



48 



Responses 

Percent 
Number (note b) 



15 
3 

11 

15 
2 

9 
9 



31 
19 



6 
16 

13 

47 
9 

34 

47 

6 

28 
28 

62 

38 
100 



37 77 

100 
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9. How beneficial do you think this workshop/seminar 

is? 

Of little or no benefit 

Of some benefit 

Of moderate benefit 

Of substantial benefit 

Of very substantial benefit 

No basis to judge 

State Assessment Programs 

10. Do you have or are you planning a statewide 
assessment program? (Check one.) 

Yes, have a program (continue) 

Yes, planning a program (continue) 

No (go to question 14) 



Number 
responding 
from 50 
(note a) 



37 



Responses 



11. Which of the following types of tests, if any, 
have been or will be used extensively in your 
statewide assessment program? (Check one or 
more. ) 



Standardized norm-referenced tests (e.g., 
the Stanford Achievement Test (SAT), the 
Wechsler Intelligence Scale for Children 
(WISC), etc.) 



Criterion-referenced tests (CRT) (e.g., 
educational objectives stated in beha- 
vioral or performance terms) 

Other tests (please specify) 

None 



50 



47 



Number 



Percent 
(note b) 



1 


3 


3 


8 


14 


38 


14 


38 


5 


14 


0 


0 




d/100 


40 


80 


7 


14 


3 


6 



19 

36 
14 
0 



100 



40 

77 
30 
0 
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Nunbttc 
responding 

from 50 
(note a) 



Responses 



12. Which of the following sources have provided or 
are scheduled to provide forael technical assist- 
ance for your statewide assessnent progi^am? 

National Assessment of Educational Progress 

National Institute of Education (NIE) 

Office of Education 

Center for the Study of Evaluation at the 

University of California at Los Angeles 

(UCLA) 

Other NIE'Sponsored centers for educational 

research and evaluation 

Other colleges and universities (specify) 

National commercial testing services (specify) 

Consultants 

Other (specify) 

None of the above 

13. Do you intend to compare your statewide assess- 
ment results with interstate and national as- 
sessment data? 

Yes, NABP data 



Yes, other interstate and national assess- 
ment data 

No 
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47 



percent 
Number (note b) 



23 
0 
4 



3 
12 
16 
28 
9 
3 



18 

0 

29 



50 
0 
9 

13 

7 
26 
35 
61 
20 

7 



38 

0 
62 
100 



52 
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Number of 
projected 
responses 
from 8941 
f note a ) 



pog ponses 
-mllb^r fercenc 



( note a ) (note^) 



^ifh the National Assess- 
Ho„ familiar are Vf p'^^Je's^gMjAEP) project 
„ent of Educational Progress V ^^^^ ^^^^ 

sDOnsored by the O^^^'^^ "Lgsion of the States? 
^^rough the education Co„„ssion^^ expresses 

jSr^.a^fliliftrwr^ -EP.) (note c) 
Have little or no information 

..miliar with the purpose and major objectives 
"pamiliar with .the project methods and 

specific objectives 

Head some project reports and critiques 
Head a substantial number of project re- 
ports and critiques 

conducted -"^^rterill To^ettr^ine^^^' 
— rts-utflitrin^rorr^aisessment program 



8242 



4412 
1985 

506 
917 

27 5 
147 



54 
24 

6 
11 



_2 
100 



,f you have read -/.^ ^brecttvrLoJ lets! ' ' 
s^^rvey results ^nd/or object iv i^a^ning 
rhich'of the following l^H^^^l, l^,,, skip 
areas are you f^^^^^f^^^^^ "our answer by 
^clircrnrthe-applopfiate rLponses. , 

Science 
writing 
Citizenship 

Reading 

Literature 

Music 

social studies 



2222 



783 
438 
626 
1658 
283 
477 
842 



Note: 



If you have c^^ecked the a t ponse^in^ 
auestion 1 ^^ove indicating thy^^^.^^^ 
"=^"f local ed ition agency (LEA), con- 
^n'ue!^ r ot, go to question 7. 



35 

20 

28 

75 
13 
21 
38 
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education program? (Ch^rk thZ ^ 

whir'h K««4. ^y*-***"*^ iv^necK the one response 

which best expresses your position.) 

Little or no utility 

A limited degree of utility 

A moderate degree of utility 

A high degree of utility 

Very high degree of utility 

No (go to question 7) 

^es (continue) 

Local policy planning 

Local assessment planning and evaluation 

— '"n.'nrairi;a?u%%'°o'n^ ^"^ P^- 

, Resource allocation 

Educational accountability 

LEA assessment comparisons 

Internal LEA assessment comparisons 

Program evaluations 

Diagnostic and/or individual needs a<;c^ce 

ment methodology neeas assess- 



Number of 
projected 
responses 
from 8941 
(note a) 



147 



146 



104 



Responses 
Number Percent 
( note a ) (note b) 



1 
40 
72 
13 
21 



104 
42 



Other (please specify) 



21 
63 

59 
0 

11 
12 
34 
70 

32 
0 



1 
27 
49 
9 

. 14 
100 

71 
29 
100 



20 
61 

57 
0 
11 
12 
33 
67 

31 
0 



56 



47 



ERIC 



APPENDIX II 



APPENDIX II 



Number of 
projected 
responses 
from 8941 
(note a) 



Responses 



Number Percent 
(note a) ( note b ) 



What ways would you suggest to improve utility 
of NAEP studies to you? (Check all that apply.) 

None 

Wider dissemination of NAEP studies 



104 



Technical guidance to initiate comparable 
State assessments 

Sro?ller geographic breakdowns, such as by 
State or county 

More detailed demographic breakdowns 

Relate NAEP data to OE-ESEA (Elementary and 
Secondary Education Act) program evalua- 
tions 

Increase the scope of the develop of 

criterion-referenced tests (note d) 

Increase the range of age assessments 

Improve the quality of the criterion- 
referenced tests (e.g., improve valid- 
ity, reduce biases and error* etc.) 

Other (please specify) 



18 
32 

12 

68 
37 

42 

55 
14 

38 
0 



17 

31 

12 

65 
36 

40 

53 
13 

37 
0 



Local Education Agency Te sting Programs 

7. Do you have or are you planning an LEA-wide 
testing program? (Check one.) 

No 

Yes, have a program 

Yes, planning a piogram 



8429 



1384 
5909 
1140 



16 
70 
14 
100 



8 DO you have or are you planning an LEA-wide 
assessment program which involves more testing 
than the regular district testing programs or 
involves more than the State assessment program, 
if one exists? 

No 



Yes, have a program 
Yes, planning a program 



8447 



4633 
1242 
2572 



55 
15 
30 
100 



If yes to either question 7 or 8, continue; if no 
to both, go to question 12. 

57 



48 



APPENDIX II 



10. 



. Which of the following types of tests do you 
usually administer or e;^pect to administer in 
either the testing or assessment program? 
(Check one or more) 

Standardized norm-referenced tests (e.g., 

the Stanford Achievement Test (SAT), the 
Wechsler Intelligence Scale for Children 
(WISC), etc.) 

Criterion-referenced tests (e.g., educa- 
tional objectives stated in behavioral 
or performance terms) 

Other tests (specify) 

Do you intend to compare your LEA-wide assess- 
ment with other LEA, State, interstate, or na- 
tional assessments? 



No 

Yes, LEA 
Yes, State 
Yes, interstate 
Yes, national 
No basis to judge 



11. Which of the following sources have provided 
or are scheduled to provide formal technical 
assistance for your statewide assessment pro- 
gram? 

National Assessment of Education Progress 

National Institute of Education (NIE) 

Office of Education 



Center for the Study of Evaluation at the 
University of California at Los Anqeles 
(UCLA) 

Other NIE-sponsored centers for educational 
research and evaluation 

Other colleges and universities (specify) 

National commercial testing services (soec- 

ify) 



Consultants 

Other (specify): 

Local school district 
State education agency 

None of the above 



Number of 
projected 
responses 
from 8941 
(note a) 



4642 



4626 



4201 
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Responses 
Number Percent 
( note a ) ( note b ) 



4171 

366 
297 



2031 
895 

1230 
13 

1328 
185 



58 



244 
30 
409 

27 
0 

291 

547 
627 

29 
612 
2158 



90 

8 
6 



44 
19 
27 

1 
29 

4 



6 
1 
10 



0 

7 

13 
15 

1 

15 
51 
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DEPARTMENT OF HEALTH, EDUCATION. AND WELFARE 

OFFICE OF THE SECRETARY 



WASHINGTON. D C. 20201 



June 3, 1976 



Mr, Gregory J. Ahart 
Director, Manpower and 

Welfare Division 
United States General 

Accounting Office 
Washington, D»C» 20548 

Dear Mr. Ahart: 

The Secretary asked that 1 respond to your request for our comments 
on your draft report eiititled "The National Assessment of Educational 
Progress: Its Results Need to be Made More Useful." The enclosed 
comments represent the tentative position of the Departmerit and are 
subjecf to reevaluation when the final version of this report is 
received. 

We appreciate the opportunity to comment on this draft report before 
Its publication^ 

Sincerely yours, 

r 

John TJ, Ypung ( 

Assistant Secretary, Comptroller 



Enclosure 
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Comments of the Department o f Hc-alth,' Kdu cation. nnd Welfare 
on cne Comptr o l ler-G^r;^ 

tveed to be Made More Useful- --fT^r^.-T^jT-jr^^^ 



DEPARTMENT OVERVIEW 

has 'conducted ^^^^ ^1 Accoanting Office (GAO) 

S^tSe issies'and lohr" '^'P'"' identifying many 

oi tne issues and problems associated with thP HpvoI^^^^ \ ^ 

B well as in its current program operations. 

While it is difficult to establish the immediate policy and curri- 
cuium relevance of NAFP T-oe,,i#-^ ^i. j . y^^j^^y ana cum- 

pros.,... ^^n.^^^l^f^.::^zr:^:i;^" 

acid materials by state ajd tlrl) It , efforts. Use of the NAEP model 

station Of the%- ::[:ate"'va'?=e='o "mI :°%:|"fo" VT'I I'l"- 
a redirection of MEP is warranted „„f,- do not believe 

its fundamental assu.p on "nfg Jin' prLcinUs"" °f 
theoreticians, substantive exnerts .^H conjunction with 

, buostantive experts and user groups has been completed. 

With reference to the recommendation in the introductory section 
users or pote = u"'r^s'o1 ^^o^^'t "rLult s^al'^t ■ ' ' V" 

fhis o^v:h^L^h^pr::^ 

"^"'"^-"i^r^^rn^orad-" -.-'^ i 
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recommend an expansion of these types of interactions with the "field;" 
however, we do not think the implication, which may be drawn by some, 
that this is a concept that is new or foreign to either NCES or NAEP 
should be allowed to stand. 



COMMENTS ON GAP RECOMMENDATIONS 

Some of the activities recommended by GAO are inter-related; 
therefore, comments are made for them as a group to avoid repetition. 

GAO RECOMMENDATION 

The Secretary, HEW, redirect the project by (1) identifying^ the 
information and other needs of decisionmakers, (2) determining t h e 
feasibility and cost effectiveness of alternative approaches for 
collecting and reporting educational assessment data and providing other 
services to satisfy those needs, and (3) dec id ing on the a s sessment 
approach to be used> It is also important to seek the views of us ers or 
potential users of project results at national. State, and local levels 
and recogaized experts in educational roanagoment7 asses^fugnt ^^nd'rc Fgarch , 

DEPARTMENT COMMENTS 

As mentionecf in the introductory comments, *we disagree with the con- 
clusion that redirection is warranted at this time! KCES is conducting 
internal studies and consultations with various agencies and individuals 
to decido on the necessity, nature or extent of any redirection. 

Also, if a new model is to be developed provision must be made for cross 
walks between present and proposed data systems, to conserve the value 
of the data that have been accumulated for a decade at the cost of 
approximately $40 million. Results of the re-examination presently 
underway through the regional meetings mechanism and more extensive NCtS- 
ASE reviews should enable us to determine whether or not NAEP should 
move to a substantially new and different design approach or attempt to 
improve upon and expand the present model and conceptual framework. In 
any event, a new model, if deemed appropriate and preferable to improving 
upon the present approach, would involve the investment of considerablr. 
time and personnel. We are attempting r:o determine the merits of various 
alternatives or options through a series of policy discussions which are 
designed to examine tlie future focus of NAEP. 
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GAP RECOMMKNDATION 

The Secretary require project management to: 

Establish continuous dialogues with the Congress, executive agenci es 
and State and local policymakers to determine their needs for dotaand 
how the National Assessment capabilities can best be used to meet jhoi^e 
needs • 



DEPARTMENT COmENTS 

We concur. During the next few years NAEP's greatest effort afid 
potential for payoff must and will be focused in this 'area. In 
January, NAEP initiated a series of regional meetings throughout the 
country involving various groups of NAEP users and potential u$ers, 
as well as national organizations and federal decision makers. These 
meetings are intended to identify decision-useful information needs. 
The present series of meetings will be completed in late May and a 
report to NCES will be prepared by this summer on the findings and 
conclusions together with NAEP's plans for activities designed to in- 
crease the utility of its data. Whi\e t\\\s is rtn iinportant first st^p, 
a continuous and systematic effort is necessary to ascertain specific 
information needs in the areas of policy making and curriculum develop- 
ment. Ihcrefore, we are planning to examine existing practices and pro-- 
cedures at various levels of government (Federal, State, and local) in 
which assessment, research or survey data are used in the decision making 
process ; 

1. to identify information needs related to educational 
decision making at various levels, with particular 
emphasis at the federal level 

2. to identify information and other needs for curriculum 
development and improvement 

The purpose of such an analysis would be to probe in depth the wcys 
in wiiich specific information is actually used at various levels of 
decision making; to identify specific data that are needed for certain 
selected kinds of decision making; and to determine the most effective 
ways of cstablishinj: and sustaining dinloguc with different user c^roups . 
Such an effort would quicken the pace of NAEP's progress toward pro- 
viding deci.?ion- and curriculum-relevant data and extend the uscfuin.-ss 
of procenL data. 



64 

55 



APPENDIX III APPENDIX III 

GAG RKCOMMtiNDATION 

The Secretary require project management to: 

Continue and increase nevly initiated efforts to interpret project data 
so as to reduce the heavy reliance on ot her s for data inter p retation . 

DEPARTMENT COMMENTS 

We concur. Although we recognize that interpretive reporting by NAEP 
may pose problems in terms of access to certain populat ion subgroups 
or institutions, NAEP plans to continue and extend its efforts in this 
area. During the last year, NAEP has -published the first two 
interpretive reports ever prepared by its staff. Previous policy, 
which caused NAEP to refrain from interpretation of its data, was 
based upon concerns associated with accessibility to various popu- 
lations and NAEP's ability to retain enough expertise to analyze 
the implications of all the assessment areas • In addition, the 
various groups that participated in the early exploration concerning 
the feasibility of a National Assessment, expressly objected to any 
possibility for analyzing or reporting any type of State or local 
school system level comparisons. 

As NAEP obtains data on additional background variables, there will 
be more opportunities for data analysis and interpretation, and 
we will continue to encourage NAEP*s increased effor': in this area. 
The constra^int we envision in the future is primarily based upon 
NAEP's inability to retain sufficient numbers of experts to permit 
NAEP to interpret all of its findings. Thus NOES would intend to 
encourage others (researchers, etc.) to confrinue to examine and re- 
port on NAEP data from a variety of perspectives in addition to 
strengthening its present analysis capability. 

GAP RECOMMENDATI ON 

The Secretary require project management to ; 

Provide for comparison of its test results wlrAi perform ance standards 
by Rivinn greater emphasis to developing th e procedures needed to 
compare those results to eencrally acceptable non-KedornI stand.^rds. 

DEPARTMI-NT CO:-;MHNTS 

So far NAI-:r*s approach to this problem ha. been to use national or 
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regional per fo rr.nncc results as a ^ sLnndard" for comparisoi^s among 
various groupc. and to provide baseline i,easures for comparisons over 
time. In .iddition, since the NAEP sample is not large enough to provide 
direct estimates .t the Stnte level or below, NAEP plans to construct 

derived estimaLos" for States, cities, districts and schools by relating 
certain mensured der,,oprnphic characteristics to achievement levels. These 
est -nates cnn serve as "standards" for those who use NAEP materials at 
the tate or local level and desire an external criterion against which 
their results cnn be compared. This approach will continue to yield . 
useful information until valid State or local level standards can be 
developed. In the meantime we will explore this problem, including the 
procedures and technical considerations associated with achieving com- 
parisons with other standards, more extensively with NAEP staff. We will 
encourage eiforts to provide the technical and methodological guidance • 

T^^L J t"'' ^^"^^^ '° "'^'^^ th^i'^ comparisons 

or judgments about appropriate standards. 

G RECOMMENDATION 



The Secretary require project management to: 

Improve its co.T.muni cat io n and cooperation with thP National Insti- 
ja^eort£ucanon_jmd^^ researchers to facilitate " 



DEPARTMENT CO MMENTS 

^nfor^.r' -^^-^tion to both formal and 

informal re la t lonsh : w,ta rhe National Institute of Education (NIE) 

and other Federal age cu s. Through the Assistant Secretary for Educa- 

lon s office, it vUI be possible to establish and maintain cert"n 
formal communication networks, since the Conmissioner of Education 
the Director o NIE. and the^ Ad.i„ ist ra tor of NCES meet regu a ly ^ith 
tion devices wUl b. devLpeo ^s i'l. communica- 

As NAEP becomes a richer source of data on input as well as outct 
measures, individual researchers and organizations wi in ,°a "^Iv 

Tin.'- - - ^^^earci-^.o^;^ 
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1. to complete a background variables data collection strategy 

2. to conduct a study of the relationship of Career and 
Occupational Development achievement data to background 
fac torii 

3. to conduct a research study to develop composite sampling 
errors 

4. to conduct an index of basic skills study 

In addition, five or six Federal agencies are presently exploring NAEP 
materials and capabilities for possible use in their prograir develop- 
ment or evaluation efforts. 

To encourage further secondary analysis of NAEP data it may be 
necessary to set asi'le some funds for the support of solicited or 
uiif^olicited analysis and interpretation of NAEP data. NCES will ex- 
plore various options pertaining to this question, but recognizes the 
limited funding possibilities presently confronting the agency. The 
current regional meetings, especially one involving federal decision 
makers and another involving professional associations, will be useful 
iij deter.riitiing how cominunicat ion and cooperation could be improved 
with researchers and research-sponsoring agencies as well as policy 
and decision makers. 

With respect to the position taken by a major test publisher that project 
results were not relevant to curriculum objectives and "too far renioved 
from the classroom level to be of use," this observation could easily 
qualify as an opinion but hardly represents an incontrovertible fact. For 
instance, a series of articles in the Ma thematics Te a cher ,^nd the 
Arithmetic Teacher , (professional journals addressed'^to teachers) discuss- 
ing the implications of assessment results for curriculum and instruction 
would seem to belie that point as a generalization at least. Clearly more 
effort must be expended i-n apprising teachers and curriculum specialists 
of NAEP's findings but we do not agree that this should be taken to ir.epu 
that they are not or cannot be useful at that level. 



GAP RECOi'^LMliiNDATION 

The Secretary require project mnnjgement to: 
Improve disseminr.t ion of project resul ts . 
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DEPAKTMENT COMMENTS 

We concur that improvement is needed in 'terms of the distribution 
of NAEP reports. As the report points out, NAEP reports are sold 
through the Government Printing Office (GPO) , and NAEP and NCES have 
minimal control over the distribution of reports. We will continue to 
seek GPO cooperation on this matter, and will explore the possibility 
of the distribut ion of the re po r t s and other publications directly by 
NAEP as one means of improving the dissemination of NAEP results. 

In addition to special studies and reports, NAEP is examining the 
feasibility of preparing special bulletins or news releases indicating 
assessment re suits at nat ional or regional levels and related infor- 
mation. In this way, targeted reports to State boards- of education, 
legislators and the like can be related to local or area level con- 
siderations that may prompt further application of NAEP results to 
educational planning and decisionmaking. 

GAO RECOMMENDATIO N 

The Secretary urge project raanaaenent to : 

Continue developine and refining objective-referenc ed tests - tests which 
give results for a group in terns of pre-de tc rmined^edu ca t ion a 1 objectives. 

DEPARTMENT C0>LMH:NTS 

- ■ t 

We concur. NAEP will continue to develop and refine the procedures for 
these acLivlLies. Several research studies, for which NAEP is seeking 
outside fuuding, could significantly strengthen NAEP's contribution in 
this area — objective and criterion-referenced testing. 

GAO RECOMMENDATI ON 

The Secretary urge project management to: 

Increase i ts technical assistance to local educational agencies re ca rd - 

inp, Its c once pt;:;, nnt oria ls, and nrthods v here nocsThln within th^ 

avrjilablc iundiii-. " ' — ^ 



DE PA RTiM R NT C O:-! ME N T S 
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We concur and will attempt to increase technical assistance to LEA*s 
within available funding. 



GAP RECOMMENDATION 

The Secretary urge project management to : 

Increase its efforts to serve Federal ay*^encies and others by providin g 
special studies for those who need and may be willing to finance them. 
The project should pursue this approach by mor adequately informing 
potential users of its capabilities in this ar-ja. 

DEPARTMENT COMMENTS 

We concur. NCES will encourage NAEP to increase its efforts to 
provide special studies for various agencies and organizations. In 
addition, NCES nnd NAEP will expand their efforts to inform govern- 
ment agencies concerning i\AEP methods, materials and findings and at- 
tempt to address specific data needs of the various agencies. Special 
conferences, staff involvenent in reviews of draft NAEP reports and 
siuiiidr activities snouia increase the awareness and involvement of 
other agencies in the potential application of NAEP results to their 
data needs. 
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Education Commissinn of the States 




300 LINCOLN lOwetl . ,»eO LINCOLN StRClT 
(XI) 893.5J00 . DCNVCR. COLORADO B0103 



••^Pril 27, 1976 



Mr Gregory J. Ahart . Director 

Ui cd States General Accounting Office 

iv.ishington. D.C. 20548 

Ucar Mr. Ahart: 



have enclosed a response to the JecomiendaSn, "1'^/°''' '"-quest, we 
wc concur in general with many of thrrP n ^ '^'"'^^^ '•-P°"- IVhile 

.at.on about recent AssessS act IrtiesXt ' '^"'^ ^''^^ 

XO... If any questions arise, call ZlH TnsZ'll j3j3r89LlJoS^^ " 



Sincerely, 



J Roy li, Forbes ' 
j f^iroctor 

enc , 



/V4r,0/V/(t ASSBSSMENT OF EDlfCA TIONAL PROGRESS 
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April 27, 1976 

Resnon.- to THE NATIONAL ASSSSSf^NT OF EDUCATIONAL PROGRESS? 

The General Accounting Office correctly identifies the long-range goal of 
National Assessn^nt as providing inforn^tion useful to educational decision- 
makers and practitioners in identifying problems, setting priorities, and deter- 
^ning progress, and the Office justifiably criticizes the Assessment for having 
attained only limited success for this g.al. Ho-ever. the achievement of long- 
range goals tyi-icauy requires extended periods of time. For this reason, the 
folWng interim goals, which were designed to measure p«>gress toward the 
long-range goal, provide more useful criteria against which to measure the 
progress of the Assessment: 

. X. TO measure change tak:ng place in selected aspects of the educational 
attainsient of young Airericans. 

2. To make available on a continuing basis comprehensive data on the 
educational attainments of young Americans. 

3. To utilize the capabilities of the National Assessment organization 
to conduct special interest probes into selected areas of educational 
attainments . 

4. To provide data, analyses, and reports understandable to, interpretable 
by and responsive to the needs of a variety of audiences. 

5. To encourage and facilitate interpretive studies of National Assessment 
data, thereby generating implications useful to educational practi- 
tioners and decisionrrakcars , 
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6. To facilitate the use of National Assessmerct *s technology at state 
and local levels as ^propriate. 

7. To continue to develop* test, and refine the technologies necessary for 
gathering and analyzing National Assessment achievement data. 

8« To conduct an ongoing program of research and operational study neces- 
sa-y for the resolution of problems and refinement of the NatiwK?.l 
Assessment model. 

Althou^ there were certainly problems , the Assessment has attained a level 
of success in each of these interim goals. The long-range goal, providing 
information useful to educational decisionmakers and practitioners, is one 
which perhaps can best be evaluated after several more years after the Assess- 
ment has fulfilled its function of identifying changes that occur over time. 

Ev«n though the goal of providing useful information for decisionmakers 
and practitioners is long-term, the Assessment has made substantial progress 
toward this goal. 



[See GAO note, p. 70. ] 



The Assessment has produced reports directly relevant to policymaking. The 
report on male-female differences in achievement, the report on racial compositit 
of schools and academic achievement, the proposed report on Spanish-sumamed 
Americans, and the proposed basic skills report are all designed to address 
policy relevant quest ions. In addition. Assessment data are a primary source 
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of information for The Condition of Education , an annual rcpon; by the National 
Center for Education Statistics for the Congress. More policy relevant reports 
should be generated as the Assessment continues to generate c lange data; 
educational trend data, as educational indicators, might well be increasingly 
relevant to policy decisions. 

While the GAO report quotes a representative from a major test publisher 
as stating Assessment data are "too far removed from the classroom to be of any 
use," the Assessment has on file letters from teachers who have used Assessment 
data in the classroom. A series of surveys indicate that the overwhelming majority 
of superintendents and principals who had access to Assessment reports stated 
that the data were applicable to specific curriculum issues and usable in the 
classroom; the respondents then listed uses in both areas. A four-man committee 
from the National Council of Teachers of Mathematics found important implica- 
tions of Assessment data for mathematics education. They published articles 
based on the data in both The Arith.T.eti c Tc-^chcr ,ir.d MnthOTTir. tics Te.icher; in 
addition. The Arithmetic Teacher is currently devoting its "Using Research in 
Teaching" section to detail results of several exercises from the National 
Assessment mathematics report. While Assessment data are presently not used to 
their full potential - perhaps in part because most education practitioners are 
not au.ire of the project - they are capable of being used. But i;c have to make 
potential users aware that the data exist and encourage their U5e. 

The General Accounting Office made the following reconwiendations to make 
the project more useful to educational decision makers, researchers, and 
practitioners: 

1. That National Acsessmcnt b^. rcdivp.cied by (<^) identifying the 

informational and other needs of decision makers, (b) determining 
the feasibility av.d ccst effectiveness of alternate approaches for 
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Belccting and reportiri^ eduaaticml aSBesement data and providing 
otJier dnrxncee to Qutiksfy thoee needst, and (a) deHding on the aeeeee-- 
nept approach to be used. We concur. We have, in fact, already 
begun to identify these inforiaation needs of decisionmakers and to 
evaluate altema'cive approaches for presenting Assessment data. An 
Office of Application has been created to identify assessment-related 
information requirements of federal, state, and local education decision- 
makers. One of the first tasks of this office has been to conduct 
a series of eight future-focus conferences designed to provide input 
for the planning process. Teachers, local and state administrators, 
members of national educational organizations, government officials, 
and university researchers have all been assembled to discuss what 
kinds of data would be most valuable to them: Should the Assessment 
continue with its subject-matter orientation or should it assess basic 
skills? Should it assess by both age and grade? Should it provide 
state-level data, either by giving states an opportunity to conduct 
simultaneous assessments or by deriving a statistical estimate of the 
data? In addition, the Assessment is presently discussing a grant 
proposal with a foundation to study how research and survey data can be 
used by decisionmakers; the study would include an examination of i;ow 
economic, agricultural, and medical data have been utilized to d..term.ne 
Whether a parallel pattern of utilization can be designed to education 
data. We assume that this approach is an appropriate response to the 
recommendation, which reinforced rather than criticizes our present efforts. 

2. That the project establish continuing dialogues with the Congresc^ 
executive agencies and state and local policymakers to determine 
their needs for data and hoQ the National Assessment capabilities 
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eon beet be twed to meet those needs. Ife concur. Several recent 
Assessment reports have been released in Washington, D,C,, and aieetings 
have been scheduled ifith key Congressional leaders in an attempt to 
establish continuing dialogues with the Congress. The Office of 
^>plication and especially the future-focus conferences, both examined 
in the preceding paragraph, are exaiq>les of some work that has already 
been accomplished in determining and meeting the needs of local, 
state, and national policymakers. 

3. That National Aeeeeewent continue to initiate effort to interpret 
project data in order to reduce the heavy reliance on others for 

data interpretation. We concur. Preliminary efforts in this area, 
include A Persp'^ctive on the First Music Assessment , An Assessment of Atti- 
tudes Toward Music , Writing Mechanics, 1969-1974 , and Science Achievement: 
Racial and Regional Trends, 1969-1973 , all of which examined the implica- 
tions of the data. Tlie positive reception of these reports has encouraged 
the Assessment to continue its efforts to interpret as well as report 
the data. In addition. Assessment personnel and experts in other 
fields examine the inplications of the data for teachers, curriculum 
•'^cvelaptTs, administrators, and policymakers for their presentations 
to national conferences and professional meetings. 

4- That the project provide for comparison of its test results with 
performance standards bu either (aj developing its oun standards 
against which project test results can be compared to judge performance 
or (b) giving greater emphasis to developing the procedures needed 
to compare its test results by other meaningful standards. We concur^ 
Some exploratory work has been attempted in this area, but 
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the problem of finding acceptable standards remains. National Assess- 
ment and the National Council for the Social Studies jointly published 
an article in which desired levels of performance were contrasted 
with actual levels of performance, but we were not satisfied with the 
method. The Assessment has submitted a grant proposal to a founda- 
tion to establish the feasibility of contrasting actual performance 
with pre-established standards, but the proposal has not yet been 
accepted. Serious steps remain in the area of establishing standards 
for objective-referenced testing. National standards may be mean- 
ingless and politically unacceptable, while local standards require 
effort at a local level. The Assessment is attempting to develop 
and disseminate a methodology with which state and local education 
agencies can establish their own standards. 

5. TTiat :;ationat Assesemnt vr^vove its ooramc-dcatlon and aoop^razion 
i^th the National Institute of Education and other educational 
researchers to facilitate the possible research, interpretation, and 
application of project results. We concur. Attempts to establish 
a working relationship with the National Institute of Education have 
included inviting Institute representatives to National Assessment briefings, 
offers to exchange information and meetings between directors. The 
Assessment now works closely with several Institute contractors and will 
continue to work toward establishing lines of communication with the Institute. 
In addition, the National Science Foundation is working jointly with 
the Assessment to fund four summer research fellows to examine technical 
problems of assessment methodology. The Assessment is also negotiating 
a grant proposal with a foundation to fund validity studies for objective- 
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referenced tests by researchers from Iowa State University, the Uni- 
versity o" Indiana, the University of Colorado, and the National Assess- 
ment, la addition, the Assessment is negotiating with research groups 
planning to examine some of the unanalyzed data from the second 
writing assessment and to perform secondary analysis on data previously 
reported. These ventures are clearly only initial efforts in establish- 
ing working relationships with various educational researchers, and 
National Assessment recognizes the need to continue these efforts. 

6. That the project improve dissemination of its results, ?s8 concur. Again, 
some progress has been made in this area, but we are planning for more. 
Each year. Assessment personnel present 40 to 50 papers at national con- 
ferences and professional meetings. Our newsletter circulation, for 
example, grows by 500 names per month, now totalling 28,000 reriders . During 
the first quarter of 1976, 10,445 publications, including reports, reprints, 
objectives booklets, monographs, informal reports, brochures, and Update 
on Education were distributed. Perhaps most impressive, however, has 
been the media coverage of the reports. The report of male-female dif- 
ferences in performance, for example, was reported in over 700 newspapers 
with a total readership of S7.5 million; reports on reading and 
writing have been publicized in Time , Newsweek, U.S. News 5 World 
^, P eaders Digest , and on NBC, CBS, and ABC. In an editorial 
\A the second writing assessment, the Washington Post maintained, 
"One sign of real hope for public education is the very fact that the 
Assessment exists." 

In addition to the six recommendations designed to increase the usefulness 
of National Assessment to educational decisionmakers, researchers, and 
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practitioners, the GAO made three more general recommendations to better utilize 
the project's capabilities; 

1. That National Aaaeaament continue to develop and refine objective-- 
referenaed teata. We concur. Present work in this area includes (a) 
attempts to refine the technology by insuring more representative 
coverage for the science and mathematics assessments and (b) feasibility 
str:dies to solve problems about how to measure difficult learning 
areas like speaking and listening skills or problem-solving skill?. 

2. That National Aaaeaament increase its technical assistance to local' 

education agencies. We concur, if resources are available. As tlie GAO 
has pointed out. National Assessment has given the bulk of its technical 
assistance to state education agencies; however, little additional 
assistance could be supported through reallocating existing funds. 
Providing greater assistance to local agencies would require additional 
funding^ possibly from an alternative source. Whatever the funding 
situation. National Assessment is presently in the process of creating 
■ore definitive documents to make it easier for local education 
agencies to use Assessment materials. In addition, we are exploring 
the possibility of providing states with statistically derived 
estimates, based on regional and demographic characteristics, of 
statewide performance on Assessment items. One way of responding to 
the need of local education agencies would be to work with state 
agencies so that they, in turn, provide technical assistance to local 
education agencies. 

3. That the project increase its efforts to serve federal agencies and 
others by providing special studies for those who need and may be 
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willing to finance them. We agree that the project should pursue 
this approach by more adequately informing potential users of its 
capabilities in this area. Ongoing and proposed special studies 
include those financed or partially financed by Ri^t to Rei.a, which 
has recently requested a proposal for a third replication of their 
reading study, the Bureau of Health Education, which has expressed 
interest in partially funding an assessment of health education, and 
the Department of Defense, which has expressed interest in funding a 
study of overseas dependent schools. Special studies that have been 
provided to other agencies include special analyses for the 
Assistant Secretary of Education, the Department of Health, Education 
and Welfare, and a study of minority skills for the National Advisory 
Committee for Equal Employment Opportunity. However, we firmly agree 
that efforts in this area should be expanded. 

National Assessment has contributed to American education by advancing 
assessment technology and assisting state education agencies in applying 
assessment technology. However, as the General Accounting Office has pointed 
out, assessment results have not been as extensively utilized by educational 
decisionmakers, researchers, and practitioners as we would like them to be. 
National Assessment has sometimes been successful in providing useful informa- 
tion for educational decisionmaking, but, in general, we agree that concentrated 
effort needs to occur in this area. 

GAO note: Deleted comments pertain to material presented 
in the draft report which has not been included 
in the final report. 
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PRINCIPAL HEW OF FICIA LS 
RESPONSIBL E FOR THE ACTIVITIES 
DISCUSSED IN THIS REPORT 



Tenure of office 
From To" 



SECRETARY OF HEALTH, EDUCATION, 
AND WELFARE: 

David Mathews 
Caspar W. Weinberger 
Frank C. Carlucci (acting) 
Elliot L. Richardson 
Robert H. Finch 

ASSISTANT SECRETARY (EDUCATION): 
Virginia Y. Trotter 
Charles B. Saunders, Jr. 

(acting ) 
Sidney p. Marland, Jr. 

COMMISSIONER OF EDUCATION: 
Terrel H. Bell 
John R. Ottina 
John R. Ottina (acting) 
Sidney P. Marland, Jr. 
Terrel H. Bell (acting) 
James E . Allen , Jr . 

ASSISTANT COMMISSIONER, NATIONAL 

CENTER FOR EDUCATIONAL STATISTICS: 
Dorothy M. Gilford 
Francis C. Nassetta 
Alexander M. Mood 

ADMINISTRATOR, NATIONAL CENTER 

FOR EDUCATION STATISTICS (note a): 
Marie d. Eldridge 
Francis C. Nassetta (acting) 
Dorothy M. Gilford (acting) 



Aug . 
Feb. 
Jan . 
June 
Jan . 



1975 
1973 
1973 
1970 
1969 



June 1974 

NOV. 1973 
Nov. 1972 



June 
Aug . 
Nov . 
Dec . 
June 
May 



1974 
1973 
1972 
1970 
1970 
1969 



May 1968 
Jan. 1968 
Jan. 1965 



Jan 
Dec 
Aug 



1976 
1974 
1974 



Present 
Aug. 1975 
Feb. 1973 
Jan. 1973 
June 1970 



Present 

June 1974 
Nov. 1973 



Present 
June 1974 



Aug . 
Nov . 
Dec . 

June 



Aug , 
May 
Jan . 



1973 
1972 
1970 
1970 



1974 
1968 
1968 



Present 
Jan. 1976 
Dec. 1974 



a/As of August 1974, the responsibility for administering 
activities relating to education statistics was trans- 
ferred from the Office of Education to the Office of the 
Assistant Secretary for Education, HEW. 
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